OpenRaiser/Intern-Atlas huggingface dataset:
SPHERE: Scaling Personalized Feedback in Programming Classrooms with Structured Review of LLM Outputs
Venue: arXiv cs.HC 2024
Year: 2024

Effective personalized feedback is crucial for learning programming. However, providing personalized, real-time feedback in large programming classrooms poses significant challenges for instructors. This paper introduces SPHERE, an interactive system that leverages Large Language Models (LLMs) and structured LLM output review to scale personalized feedback for in-class coding activities. SPHERE employs two key components: an Issue Recommendation Component that identifies critical patterns in students' code and discussion, and a Feedback Review Component that uses a ``strategy-detail-verify'' approach for efficient feedback creation and verification. An in-lab, between-subject study demonstrates SPHERE's effectiveness in improving feedback quality and the overall feedback review process compared to a baseline system using off-the-shelf LLM outputs. This work contributes a novel approach to scaling personalized feedback in programming education, addressing the challenges of real-time response, issue prioritization, and large-scale personalization.
Distributed Online Life-Long Learning (DOL3) for Multi-agent Trust and Reputation Assessment in E-commerce
Venue: arXiv cs.AI 2024
Year: 2024

Trust and Reputation Assessment of service providers in citizen-focused environments like e-commerce is vital to maintain the integrity of the interactions among agents. The goals and objectives of both the service provider and service consumer agents are relevant to the goals of the respective citizens (end users). The provider agents often pursue selfish goals that can make the service quality highly volatile, contributing towards the non-stationary nature of the environment. The number of active service providers tends to change over time resulting in an open environment. This necessitates a rapid and continual assessment of the Trust and Reputation. A large number of service providers in the environment require a distributed multi-agent Trust and Reputation assessment. This paper addresses the problem of multi-agent Trust and Reputation Assessment in a non-stationary environment involving transactions between providers and consumers. In this setting, the observer agents carry out the assessment and communicate their assessed trust scores with each other over a network. We propose a novel Distributed Online Life-Long Learning (DOL3) algorithm that involves real-time rapid learning of trust and reputation scores of providers. Each observer carries out an adaptive learning and weighted fusion process combining their own assessment along with that of their neighbour in the communication network. Simulation studies reveal that the state-of-the-art methods, which usually involve training a model to assess an agent's trust and reputation, do not work well in such an environment. The simulation results show that the proposed DOL3 algorithm outperforms these methods and effectively handles the volatility in such environments. From the statistical evaluation, it is evident that DOL3 performs better compared to other models in 90% of the cases.
On the Uniqueness of Nash Equilibria in Multiagent Matrix Games
Venue: arXiv cs.GT 2024
Year: 2024

We provide a complete characterization for uniqueness of equilibria in unconstrained polymatrix games. We show that while uniqueness is natural for coordination and general polymatrix games, zero-sum games require that the dimension of the combined strategy space is even. Therefore, non-uniqueness is common in zero-sum polymatrix games. In addition, we study the impact of non-uniqueness on classical learning dynamics for multiagent systems and show that the classical methods still yield unique estimates even when there is not a unique equilibrium.
How Can We Diagnose and Treat Bias in Large Language Models for Clinical Decision-Making?
Venue: arXiv cs.AI 2024
Year: 2024

Recent advancements in Large Language Models (LLMs) have positioned them as powerful tools for clinical decision-making, with rapidly expanding applications in healthcare. However, concerns about bias remain a significant challenge in the clinical implementation of LLMs, particularly regarding gender and ethnicity. This research investigates the evaluation and mitigation of bias in LLMs applied to complex clinical cases, focusing on gender and ethnicity biases. We introduce a novel Counterfactual Patient Variations (CPV) dataset derived from the JAMA Clinical Challenge. Using this dataset, we built a framework for bias evaluation, employing both Multiple Choice Questions (MCQs) and corresponding explanations. We explore prompting with eight LLMs and fine-tuning as debiasing methods. Our findings reveal that addressing social biases in LLMs requires a multidimensional approach as mitigating gender bias can occur while introducing ethnicity biases, and that gender bias in LLM embeddings varies significantly across medical specialities. We demonstrate that evaluating both MCQ response and explanation processes is crucial, as correct responses can be based on biased \textit{reasoning}. We provide a framework for evaluating LLM bias in real-world clinical cases, offer insights into the complex nature of bias in these models, and present strategies for bias mitigation.
Efficient and Effective Algorithms for A Family of Influence Maximization Problems with A Matroid Constraint
Venue: arXiv cs.SI 2024
Year: 2024

Influence maximization (IM) is a classic problem that aims to identify a small group of critical individuals, known as seeds, who can influence the largest number of users in a social network through word-of-mouth. This problem finds important applications including viral marketing, infection detection, and misinformation containment. The conventional IM problem is typically studied with the oversimplified goal of selecting a single seed set. Many real-world scenarios call for multiple sets of seeds, particularly on social media platforms where various viral marketing campaigns need different sets of seeds to propagate effectively. To this end, previous works have formulated various IM variants, central to which is the requirement of multiple seed sets, naturally modeled as a matroid constraint. However, the current best-known solutions for these variants either offer a weak $(1/2-ε)$-approximation, or offer a $(1-1/e-ε)$-approximation algorithm that is very expensive. We propose an efficient seed selection method called AMP, an algorithm with a $(1-1/e-ε)$-approximation guarantee for this family of IM variants. To further improve efficiency, we also devise a fast implementation, called RAMP. We extensively evaluate the performance of our proposal against 6 competitors across 4 IM variants and on 7 real-world networks, demonstrating that our proposal outperforms all competitors in terms of result quality, running time, and memory usage. We have also deployed RAMP in a real industry strength application involving online gaming, where we show that our deployed solution significantly improves upon the baselines.
A Statistical Analysis of LLMs' Self-Evaluation Using Proverbs
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) such as ChatGPT, GPT-4, Claude-3, and Llama are being integrated across a variety of industries. Despite this rapid proliferation, experts are calling for caution in the interpretation and adoption of LLMs, owing to numerous associated ethical concerns. Research has also uncovered shortcomings in LLMs' reasoning and logical abilities, raising questions on the potential of LLMs as evaluation tools. In this paper, we investigate LLMs' self-evaluation capabilities on a novel proverb reasoning task. We introduce a novel proverb database consisting of 300 proverb pairs that are similar in intent but different in wordings, across topics spanning gender, wisdom, and society. We propose tests to evaluate textual consistencies as well as numerical consistencies across similar proverbs, and demonstrate the effectiveness of our method and dataset in identifying failures in LLMs' self-evaluation which in turn can highlight issues related to gender stereotypes and lack of cultural understanding in LLMs.
Chatting with Bots: AI, Speech Acts, and the Edge of Assertion
Venue: arXiv cs.CL 2024
Year: 2024

This paper addresses the question of whether large language model-powered chatbots are capable of assertion. According to what we call the Thesis of Chatbot Assertion (TCA), chatbots are the kinds of things that can assert, and at least some of the output produced by current-generation chatbots qualifies as assertion. We provide some motivation for TCA, arguing that it ought to be taken seriously and not simply dismissed. We also review recent objections to TCA, arguing that these objections are weighty. We thus confront the following dilemma: how can we do justice to both the considerations for and against TCA? We consider two influential responses to this dilemma - the first appeals to the notion of proxy-assertion; the second appeals to fictionalism - and argue that neither is satisfactory. Instead, reflecting on the ontogenesis of assertion, we argue that we need to make space for a category of proto-assertion. We then apply the category of proto-assertion to chatbots, arguing that treating chatbots as proto-assertors provides a satisfactory resolution to the dilemma of chatbot assertion.
RKadiyala at SemEval-2024 Task 8: Black-Box Word-Level Text Boundary Detection in Partially Machine Generated Texts
Venue: arXiv cs.CL 2024
Year: 2024

With increasing usage of generative models for text generation and widespread use of machine generated texts in various domains, being able to distinguish between human written and machine generated texts is a significant challenge. While existing models and proprietary systems focus on identifying whether given text is entirely human written or entirely machine generated, only a few systems provide insights at sentence or paragraph level at likelihood of being machine generated at a non reliable accuracy level, working well only for a set of domains and generators. This paper introduces few reliable approaches for the novel task of identifying which part of a given text is machine generated at a word level while comparing results from different approaches and methods. We present a comparison with proprietary systems , performance of our model on unseen domains' and generators' texts. The findings reveal significant improvements in detection accuracy along with comparison on other aspects of detection capabilities. Finally we discuss potential avenues for improvement and implications of our work. The proposed model is also well suited for detecting which parts of a text are machine generated in outputs of Instruct variants of many LLMs.
CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing
Venue: arXiv cs.LG 2024
Year: 2024

Sequential reasoning in agent systems has been significantly advanced by large language models (LLMs), yet existing approaches face limitations. Reflection-driven reasoning relies solely on knowledge in pretrained models, limiting performance in novel scenarios, while experience-assisted reasoning often depends on external experiences and lacks clear principles for selecting representative experiences. We address these limitations by proposing CoPS (Cross-Task Experience Sharing), a generalizable algorithm that enhances sequential reasoning by cross-task experience sharing and selection. In detail, CoPS leverages agents' experiences on previous tasks, selecting distribution-matched experiences via a provable pessimism-based strategy to maximize utility while minimizing risks from distribution shifts. Extensive experimental results on benchmarks like Alfworld, Webshop, and HotPotQA demonstrate that CoPS consistently outperforms state-of-the-art baselines, with superior sample efficiency suitable for resource-constrained scenarios. Theoretically, we show that the performance of our algorithm depends on both the quality of the pretrained LLM and the matching between the agent's task-dependent trial distribution and that generated by the LLM. Our work bridges the gap between existing sequential reasoning paradigms and validates the effectiveness of leveraging cross-task experiences, shedding light on the potential to improve agents' generalization and adaptability across diverse tasks. Our codes are available at $\href{https://github.com/uclaml/COPS}{\text{https://github.com/uclaml/COPS}}$.
DENOASR: Debiasing ASRs through Selective Denoising
Venue: arXiv cs.SD 2024
Year: 2024

Automatic Speech Recognition (ASR) systems have been examined and shown to exhibit biases toward particular groups of individuals, influenced by factors such as demographic traits, accents, and speech styles. Noise can disproportionately impact speakers with certain accents, dialects, or speaking styles, leading to biased error rates. In this work, we introduce a novel framework DENOASR, which is a selective denoising technique to reduce the disparity in the word error rates between the two gender groups, male and female. We find that a combination of two popular speech denoising techniques, viz. DEMUCS and LE, can be effectively used to mitigate ASR disparity without compromising their overall performance. Experiments using two state-of-the-art open-source ASRs - OpenAI WHISPER and NVIDIA NEMO - on multiple benchmark datasets, including TIE, VOX-POPULI, TEDLIUM, and FLEURS, show that there is a promising reduction in the average word error rate gap across the two gender groups. For a given dataset, the denoising is selectively applied on speech samples having speech intelligibility below a certain threshold, estimated using a small validation sample, thus ameliorating the need for large-scale human-written ground-truth transcripts. Our findings suggest that selective denoising can be an elegant approach to mitigate biases in present-day ASR systems.
NodeOP: Optimizing Node Management for Decentralized Networks
Venue: arXiv cs.DB 2024
Year: 2024

We present NodeOP, a novel framework designed to optimize the management of General Node Operators in decentralized networks. By integrating Agent-Based Modeling (ABM) with a Tendermint Byzantine Fault Tolerance (BFT)-based consensus mechanism, NodeOP addresses key challenges in task allocation, consensus formation, and system stability. Through rigorous mathematical modeling and formal optimization, NodeOP ensures stable equilibrium in node task distribution. We validate the framework via convergence analysis and performance metrics such as transaction throughput, system latency, and fault tolerance. We further demonstrate NodeOP's practical utility through two use cases: decentralized sequencer management in Layer 2 networks and off-chain payment validation. These examples underscore how NodeOP enhances validation efficiency and unlocks new revenue opportunities in large-scale decentralized environments. Our results position NodeOP as a scalable and flexible solution, significantly improving operational efficiency and economic sustainability in decentralized systems.
Efficient Scheduling of Vehicular Tasks on Edge Systems with Green Energy and Battery Storage
Venue: arXiv cs.DC 2024
Year: 2024

The autonomous vehicle industry is rapidly expanding, requiring significant computational resources for tasks like perception and decision-making. Vehicular edge computing has emerged to meet this need, utilizing roadside computational units (roadside edge servers) to support autonomous vehicles. Aligning with the trend of green cloud computing, these roadside edge servers often get energy from solar power. Additionally, each roadside computational unit is equipped with a battery for storing solar power, ensuring continuous computational operation during periods of low solar energy availability. In our research, we address the scheduling of computational tasks generated by autonomous vehicles to roadside units with power consumption proportional to the cube of the computational load of the server. Each computational task is associated with a revenue, dependent on its computational needs and deadline. Our objective is to maximize the total revenue of the system of roadside computational units. We propose an offline heuristics approach based on predicted solar energy and incoming task patterns for different time slots. Additionally, we present heuristics for real-time adaptation to varying solar energy and task patterns from predicted values for different time slots. Our comparative analysis shows that our methods outperform state-of-the-art approaches upto 40\% for real-life datasets.
RIS-Assisted THz MIMO Wireless System in the Presence of Direct Link for CV-QKD with Limited Quantum Memory
Venue: arXiv cs.IT 2024
Year: 2024

A reconfigurable intelligent surface (RIS)-aided multiple-input multiple-output (MIMO) wireless communication system is considered in this paper wherein the transmitter, Alice modulates secret keys, by using a continuous variable quantum key distribution technique to be transmitted to the receiver, Bob, which employs homodyne detection for data decoding. The data is transmitted over two paths, namely a direct path between Alice and Bob and the wireless path between them via the RIS. Transmit and receive beamsplitters are employed in the system to transform the MIMO terahertz channels into parallel single-input single-output channels. Considering an eavesdropper, Eve, to attack all the three wireless channels in the system (i.e., the direct channel, the channel between Alice and RIS, and between the RIS and Bob) but having restricted quantum memory limiting it to store the ancilla modes from either of these three wireless channels, novel expressions for the secret key rate (SKR) of the system are derived. Numerical results are presented to demonstrate the dependency of the system's performance on various system parameters. It is observed that the RIS plays a key role in increasing the SKR of the system and the transmission distance, ensuring secure communications between Alice and Bob. The significance of employing RIS is observed specifically for the case when Eve measures the ancilla modes of the channel between the RIS and Bob. Furthermore, for all such measurement scenarios, optimal angles are obtained for the phase shifts of the RIS elements to maximize the SKR for various MIMO configurations and transmission distance between Alice and Bob.
50 questions on Active Assisted Living technologies. Global edition
Venue: arXiv cs.HC 2024
Year: 2024

This booklet on Active Assisted Living (AAL) technologies has been created as part of the GoodBrother COST Action, which has run from 2020 to 2024. COST Actions are European research programs that promote collaboration across borders, uniting researchers, professionals, and institutions to address key societal challenges. GoodBrother focused on ethical and privacy concerns surrounding video and audio monitoring in care settings. The aim was to ensure that while AAL technologies help older adults and vulnerable individuals, their privacy and data protection rights remain a top priority. This booklet is designed to guide you through the role that AAL technologies play in improving the quality of life for older adults, caregivers, and people with disabilities. AAL technologies offer tools for those facing cognitive or physical challenges. They can enhance independence, assist with daily routines, and promote a safer living environment. However, the rise of these technologies also brings important questions about data protection and user autonomy. This resource is intended for a wide audience, including end users, caregivers, healthcare professionals, and policymakers. It provides practical guidance on integrating AAL technologies into care settings while safeguarding privacy and ensuring ethical use. The insights offered here aim to empower users and caregivers to make informed choices that enhance both the quality of care and respect for personal autonomy.
Co-training partial domain adaptation networks for industrial Fault Diagnosis
Venue: arXiv cs.LG 2024
Year: 2024

The partial domain adaptation (PDA) challenge is a prevalent issue in industrial fault diagnosis. Drawing inspiration from traditional classification settings where such partial challenge is not a concern, we propose a novel PDA framework called Interactive Residual Domain Adaptation Networks (IRDAN), which introduces domain-wise models for each domain to provide a new perspective for the PDA challenge. Each domain-wise model is equipped with a residual domain adaptation (RDA) block to mitigate the ADP problem. Additionally, we introduce a confident information flow via an interactive learning strategy, training the modules of IRDAN sequentially to avoid cross-interference. We also establish a reliable stopping criterion for selecting the best-performing model, ensuring practical usability in real-world applications. Experiments have demonstrated the superior performance of the proposed IRDAN.
Correct after Answer: Enhancing Multi-Span Question Answering with Post-Processing Method
Venue: arXiv cs.CL 2024
Year: 2024

Multi-Span Question Answering (MSQA) requires models to extract one or multiple answer spans from a given context to answer a question. Prior work mainly focuses on designing specific methods or applying heuristic strategies to encourage models to predict more correct predictions. However, these models are trained on gold answers and fail to consider the incorrect predictions. Through a statistical analysis, we observe that models with stronger abilities do not predict less incorrect predictions compared with other models. In this work, we propose Answering-Classifying-Correcting (ACC) framework, which employs a post-processing strategy to handle incorrect predictions. Specifically, the ACC framework first introduces a classifier to classify the predictions into three types and exclude "wrong predictions", then introduces a corrector to modify "partially correct predictions". Experiments on several MSQA datasets show that ACC framework significantly improves the Exact Match (EM) scores, and further analysis demostrates that ACC framework efficiently reduces the number of incorrect predictions, improving the quality of predictions.
Safe Load Balancing in Software-Defined-Networking
Venue: arXiv cs.NI 2024
Year: 2024

High performance, reliability and safety are crucial properties of any Software-Defined-Networking (SDN) system. Although the use of Deep Reinforcement Learning (DRL) algorithms has been widely studied to improve performance, their practical applications are still limited as they fail to ensure safe operations in exploration and decision-making. To fill this gap, we explore the design of a Control Barrier Function (CBF) on top of Deep Reinforcement Learning (DRL) algorithms for load-balancing. We show that our DRL-CBF approach is capable of meeting safety requirements during training and testing while achieving near-optimal performance in testing. We provide results using two simulators: a flow-based simulator, which is used for proof-of-concept and benchmarking, and a packet-based simulator that implements real protocols and scheduling. Thanks to the flow-based simulator, we compared the performance against the optimal policy, solving a Non Linear Programming (NLP) problem with the SCIP solver. Furthermore, we showed that pre-trained models in the flow-based simulator, which is faster, can be transferred to the packet simulator, which is slower but more accurate, with some fine-tuning. Overall, the results suggest that near-optimal Quality-of-Service (QoS) performance in terms of end-to-end delay can be achieved while safety requirements related to link capacity constraints are guaranteed. In the packet-based simulator, we also show that our DRL-CBF algorithms outperform non-RL baseline algorithms. When the models are fine-tuned over a few episodes, we achieved smoother QoS and safety in training, and similar performance in testing compared to the case where models have been trained from scratch.