OpenRaiser/Intern-Atlas huggingface dataset:
Navigating Ethics and Power Dynamics through Participant-Designer Journey Mapping
Venue: arXiv cs.HC 2024
Year: 2024

As Digital Transformation and innovation driven by Information and Communication Technology (ICT) continue to mark the evolution of society, ethics emerges as a central concern not only in terms of the outcomes and implications of technological systems but also throughout the activities carried out through the development of those systems. Power dynamics have been identified as a recurrent ethical challenge in the design and development process. As designers, participants, and project stakeholders engage in the process, potential conflicts, power imbalances, and ethical challenges emerge. This requires that awareness is raised on these imbalances and that teams proactively act on them. To address this issue, we propose the Participant-Designer Journey Map (PDJM), a tool to assist designers in conducting an ethical design process, aware of power imbalances. The proposal for the PDJM was evaluated, based on a set of criteria derived from the literature, by three design professionals, against nine other alternative tools. The PDJM was identified as the tool with the most potential to facilitate a structured approach to navigating ethical dilemmas, particularly those related to power dynamics.
SRAS: Self-governed Remote Attestation Scheme for Multi-party Collaboration
Venue: arXiv cs.CR 2024
Year: 2024

Trusted Execution Environments (TEEs), such as Intel Software Guard Extensions (SGX), ensure the confidentiality and integrity of user applications when using cloud computing resources. However, in the multi-party cloud computing scenario, how to select a Relying Party to verify the TEE of each party and avoid leaking sensitive data to each other remains an open question. In this paper, we propose SRAS, an open self-governed remote attestation scheme with attestation and verification functions for verifying the trustworthiness of TEEs and computing assets, achieving decentralized unified trusted attestation and verification platform for multi-party cloud users. In SRAS, we design a Relying Party enclave, which can form a virtual verifiable network, capable of local verification on behalf of other participants relying parties without leaking sensitive data to others. We provide an open-source prototype implementation of SRAS to facilitate the adoption of this technology by cloud users or developers.
Design of a Health Monitoring System for a Planetary Exploration Rover
Venue: arXiv cs.RO 2024
Year: 2024

It is generally considered that a trustworthy autonomous planetary exploration rover must be able to operate safely and effectively within its environment. Central to trustworthy operation is the ability for the rover to recognise and diagnose abnormal behaviours during its operation. Failure to diagnose faulty behaviour could lead to degraded performance or an unplanned halt in operation. This work investigates a health monitoring method that can be used to improve the capabilities of a fault detection system for a planetary exploration rover. A suite of four metrics, named 'rover vitals', are evaluated as indicators of degradation in the rover's performance. These vitals are combined to give an overall estimate of the rover's 'health'. By comparing the behaviour of a faulty real system with a non-faulty observer, residuals are generated in terms of two high-level metrics: heading and velocity. Adaptive thresholds are applied to the residuals to enable the detection of faulty behaviour, where the adaptive thresholds are informed by the rover's perceived health. Simulation experiments carried out in MATLAB showed that the proposed health monitoring and fault detection methodology can detect high-risk faults in both the sensors and actuators of the rover.
Decoding Political Polarization in Social Media Interactions
Venue: arXiv cs.SI 2024
Year: 2024

Social media platforms significantly influence ideological divisions by enabling users to select information that aligns with their beliefs and avoid opposing viewpoints. Analyzing approximately 47 million Facebook posts, this study investigates the interactions of around 170 million users with news pages, revealing distinct patterns based on political orientations. While users generally prefer content that reflects their political biases, the extent of engagement varies even among individuals with similar ideological leanings. Specifically, political biases heavily influence commenting behaviors, particularly among users leaning towards the center-left and the right. Conversely, the 'likes' from center-left and centrist users are more indicative of their political affiliations. This research illuminates the complex relationship between social media behavior and political polarization, offering new insights into the manifestation of ideological divisions online.
On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) have showcased remarkable capabilities in various Natural Language Processing tasks. For automatic open-domain dialogue evaluation in particular, LLMs have been seamlessly integrated into evaluation frameworks, and together with human evaluation, compose the backbone of most evaluations. However, existing evaluation benchmarks often rely on outdated datasets and evaluate aspects like Fluency and Relevance, which fail to adequately capture the capabilities and limitations of state-of-the-art chatbot models. This paper critically examines current evaluation benchmarks, highlighting that the use of older response generators and quality aspects fail to accurately reflect modern chatbot capabilities. A small annotation experiment on a recent LLM-generated dataset (SODA) reveals that LLM evaluators such as GPT-4 struggle to detect actual deficiencies in dialogues generated by current LLM chatbots.
Implicit Hypersurface Approximation Capacity in Deep ReLU Networks
Venue: arXiv cs.LG 2024
Year: 2024

We develop a geometric approximation theory for deep feed-forward neural networks with ReLU activations. Given a $d$-dimensional hypersurface in $\mathbb{R}^{d+1}$ represented as the graph of a $C^2$-function $ϕ$, we show that a deep fully-connected ReLU network of width $d+1$ can implicitly construct an approximation as its zero contour with a precision bound depending on the number of layers. This result is directly applicable to the binary classification setting where the sign of the network is trained as a classifier, with the network's zero contour as a decision boundary. Our proof is constructive and relies on the geometrical structure of ReLU layers provided in [doi:10.48550/arXiv.2310.03482]. Inspired by this geometrical description, we define a new equivalent network architecture that is easier to interpret geometrically, where the action of each hidden layer is a projection onto a polyhedral cone derived from the layer's parameters. By repeatedly adding such layers, with parameters chosen such that we project small parts of the graph of $ϕ$ from the outside in, we, in a controlled way, construct a network that implicitly approximates the graph over a ball of radius $R$. The accuracy of this construction is controlled by a discretization parameter $δ$ and we show that the tolerance in the resulting error bound scales as $(d-1)R^{3/2}δ^{1/2}$ and the required number of layers is of order $d\big(\frac{32R}δ\big)^{\frac{d+1}{2}}$.
TartuNLP @ AXOLOTL-24: Leveraging Classifier Output for New Sense Detection in Lexical Semantics
Venue: arXiv cs.CL 2024
Year: 2024

We present our submission to the AXOLOTL-24 shared task. The shared task comprises two subtasks: identifying new senses that words gain with time (when comparing newer and older time periods) and producing the definitions for the identified new senses. We implemented a conceptually simple and computationally inexpensive solution to both subtasks. We trained adapter-based binary classification models to match glosses with usage examples and leveraged the probability output of the models to identify novel senses. The same models were used to match examples of novel sense usages with Wiktionary definitions. Our submission attained third place on the first subtask and the first place on the second subtask.
The Solution for the GAIIC2024 RGB-TIR object detection Challenge
Venue: arXiv cs.CV 2024
Year: 2024

This report introduces a solution to The task of RGB-TIR object detection from the perspective of unmanned aerial vehicles. Unlike traditional object detection methods, RGB-TIR object detection aims to utilize both RGB and TIR images for complementary information during detection. The challenges of RGB-TIR object detection from the perspective of unmanned aerial vehicles include highly complex image backgrounds, frequent changes in lighting, and uncalibrated RGB-TIR image pairs. To address these challenges at the model level, we utilized a lightweight YOLOv9 model with extended multi-level auxiliary branches that enhance the model's robustness, making it more suitable for practical applications in unmanned aerial vehicle scenarios. For image fusion in RGB-TIR detection, we incorporated a fusion module into the backbone network to fuse images at the feature level, implicitly addressing calibration issues. Our proposed method achieved an mAP score of 0.516 and 0.543 on A and B benchmarks respectively while maintaining the highest inference speed among all models.
Entity-Level Sentiment: More than the Sum of Its Parts
Venue: arXiv cs.CL 2024
Year: 2024

In sentiment analysis of longer texts, there may be a variety of topics discussed, of entities mentioned, and of sentiments expressed regarding each entity. We find a lack of studies exploring how such texts express their sentiment towards each entity of interest, and how these sentiments can be modelled. In order to better understand how sentiment regarding persons and organizations (each entity in our scope) is expressed in longer texts, we have collected a dataset of expert annotations where the overall sentiment regarding each entity is identified, together with the sentence-level sentiment for these entities separately. We show that the reader's perceived sentiment regarding an entity often differs from an arithmetic aggregation of sentiments at the sentence level. Only 70\% of the positive and 55\% of the negative entities receive a correct overall sentiment label when we aggregate the (human-annotated) sentiment labels for the sentences where the entity is mentioned. Our dataset reveals the complexity of entity-specific sentiment in longer texts, and allows for more precise modelling and evaluation of such sentiment expressions.
Near-optimal Size Linear Sketches for Hypergraph Cut Sparsifiers
Venue: arXiv cs.DS 2024
Year: 2024

A $(1 \pm ε)$-sparsifier of a hypergraph $G(V,E)$ is a (weighted) subgraph that preserves the value of every cut to within a $(1 \pm ε)$-factor. It is known that every hypergraph with $n$ vertices admits a $(1 \pm ε)$-sparsifier with $\tilde{O}(n/ε^2)$ hyperedges. In this work, we explore the task of building such a sparsifier by using only linear measurements (a \emph{linear sketch}) over the hyperedges of $G$, and provide nearly-matching upper and lower bounds for this task. Specifically, we show that there is a randomized linear sketch of size $\widetilde{O}(n r \log(m) / ε^2)$ bits which with high probability contains sufficient information to recover a $(1 \pm ε)$ cut-sparsifier with $\tilde{O}(n/ε^2)$ hyperedges for any hypergraph with at most $m$ edges each of which has arity bounded by $r$. This immediately gives a dynamic streaming algorithm for hypergraph cut sparsification with an identical space complexity, improving on the previous best known bound of $\widetilde{O}(n r^2 \log^4(m) / ε^2)$ bits of space (Guha, McGregor, and Tench, PODS 2015). We complement our algorithmic result above with a nearly-matching lower bound. We show that for every $ε\in (0,1)$, one needs $Ω(nr \log(m/n) / \log(n))$ bits to construct a $(1 \pm ε)$-sparsifier via linear sketching, thus showing that our linear sketch achieves an optimal dependence on both $r$ and $\log(m)$.
TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Classical Chinese is a gateway to the rich heritage and wisdom of ancient China, yet its complexities pose formidable comprehension barriers for most modern people without specialized knowledge. While Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), they struggle with Classical Chinese Understanding (CCU), especially in data-demanding and knowledge-intensive tasks. In response to this dilemma, we propose \textbf{TongGu} (mean understanding ancient and modern), the first CCU-specific LLM, underpinned by three core contributions. First, we construct a two-stage instruction-tuning dataset ACCN-INS derived from rich classical Chinese corpora, aiming to unlock the full CCU potential of LLMs. Second, we propose Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting, enabling TongGu to acquire new capabilities while preserving its foundational knowledge. Third, we present a CCU Retrieval-Augmented Generation (CCU-RAG) technique to reduce hallucinations based on knowledge-grounding. Extensive experiments across 24 diverse CCU tasks validate TongGu's superior ability, underscoring the effectiveness of RAT and CCU-RAG. The model and dataset are available at \url{https://github.com/SCUT-DLVCLab/TongGu-LLM}.
Diverse and Fine-Grained Instruction-Following Ability Exploration with Synthetic Data
Venue: arXiv cs.AI 2024
Year: 2024

Instruction-following is particularly crucial for large language models (LLMs) to support diverse user requests. While existing work has made progress in aligning LLMs with human preferences, evaluating their capabilities on instruction following remains a challenge due to complexity and diversity of real-world user instructions. While existing evaluation methods focus on general skills, they suffer from two main shortcomings, i.e., lack of fine-grained task-level evaluation and reliance on singular instruction expression. To address these problems, this paper introduces DINGO, a fine-grained and diverse instruction-following evaluation dataset that has two main advantages: (1) DINGO is based on a manual annotated, fine-grained and multi-level category tree with 130 nodes derived from real-world user requests; (2) DINGO includes diverse instructions, generated by both GPT-4 and human experts. Through extensive experiments, we demonstrate that DINGO can not only provide more challenging and comprehensive evaluation for LLMs, but also provide task-level fine-grained directions to further improve LLMs.
Efficient Maximal Frequent Group Enumeration in Temporal Bipartite Graphs
Venue: arXiv cs.DB 2024
Year: 2024

Cohesive subgraph mining is a fundamental problem in bipartite graph analysis. In reality, relationships between two types of entities often occur at some specific timestamps, which can be modeled as a temporal bipartite graph. However, the temporal information is widely neglected by previous studies. Moreover, directly extending the existing models may fail to find some critical groups in temporal bipartite graphs, which appear in a unilateral (i.e., one-layer) form. To fill the gap, in this paper, we propose a novel model, called maximal λ-frequency group (MFG). Given a temporal bipartite graph G=(U,V,E), a vertex set V_S \subseteq V is an MFG if i) there are no less than λtimestamps, at each of which V_S can form a (t_U,t_V)-biclique with some vertices in U at the corresponding snapshot, and ii) it is maximal. To solve the problem, a filter-and-verification (FilterV) method is proposed based on the Bron-Kerbosch framework, incorporating novel filtering techniques to reduce the search space and array-based strategy to accelerate the frequency and maximality verification. Nevertheless, the cost of frequency verification in each valid candidate set computation and maximality check could limit the scalability of FilterV to larger graphs. Therefore, we further develop a novel verification-free (VFree) approach by leveraging the advanced dynamic counting structure proposed. Theoretically, we prove that VFree can reduce the cost of each valid candidate set computation in FilterV by a factor of O(|V|). Furthermore, VFree can avoid the explicit maximality verification because of the developed search paradigm. Finally, comprehensive experiments on 15 real-world graphs are conducted to demonstrate the efficiency and effectiveness of the proposed techniques and model.
Meta-prompting Optimized Retrieval-augmented Generation
Venue: arXiv cs.CL 2024
Year: 2024

Retrieval-augmented generation resorts to content retrieved from external sources in order to leverage the performance of large language models in downstream tasks. The excessive volume of retrieved content, the possible dispersion of its parts, or their out of focus range may happen nevertheless to eventually have a detrimental rather than an incremental effect. To mitigate this issue and improve retrieval-augmented generation, we propose a method to refine the retrieved content before it is included in the prompt by resorting to meta-prompting optimization. Put to empirical test with the demanding multi-hop question answering task from the StrategyQA dataset, the evaluation results indicate that this method outperforms a similar retrieval-augmented system but without this method by over 30%.
Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Low sample efficiency is an enduring challenge of reinforcement learning (RL). With the advent of versatile large language models (LLMs), recent works impart common-sense knowledge to accelerate policy learning for RL processes. However, we note that such guidance is often tailored for one specific task but loses generalizability. In this paper, we introduce a framework that harnesses LLMs to extract background knowledge of an environment, which contains general understandings of the entire environment, making various downstream RL tasks benefit from one-time knowledge representation. We ground LLMs by feeding a few pre-collected experiences and requesting them to delineate background knowledge of the environment. Afterward, we represent the output knowledge as potential functions for potential-based reward shaping, which has a good property for maintaining policy optimality from task rewards. We instantiate three variants to prompt LLMs for background knowledge, including writing code, annotating preferences, and assigning goals. Our experiments show that these methods achieve significant sample efficiency improvements in a spectrum of downstream tasks from Minigrid and Crafter domains.
ROER: Regularized Optimal Experience Replay
Venue: arXiv cs.LG 2024
Year: 2024

Experience replay serves as a key component in the success of online reinforcement learning (RL). Prioritized experience replay (PER) reweights experiences by the temporal difference (TD) error empirically enhancing the performance. However, few works have explored the motivation of using TD error. In this work, we provide an alternative perspective on TD-error-based reweighting. We show the connections between the experience prioritization and occupancy optimization. By using a regularized RL objective with $f-$divergence regularizer and employing its dual form, we show that an optimal solution to the objective is obtained by shifting the distribution of off-policy data in the replay buffer towards the on-policy optimal distribution using TD-error-based occupancy ratios. Our derivation results in a new pipeline of TD error prioritization. We specifically explore the KL divergence as the regularizer and obtain a new form of prioritization scheme, the regularized optimal experience replay (ROER). We evaluate the proposed prioritization scheme with the Soft Actor-Critic (SAC) algorithm in continuous control MuJoCo and DM Control benchmark tasks where our proposed scheme outperforms baselines in 6 out of 11 tasks while the results of the rest match with or do not deviate far from the baselines. Further, using pretraining, ROER achieves noticeable improvement on difficult Antmaze environment where baselines fail, showing applicability to offline-to-online fine-tuning. Code is available at \url{https://github.com/XavierChanglingLi/Regularized-Optimal-Experience-Replay}.
A Critical Assessment of Interpretable and Explainable Machine Learning for Intrusion Detection
Venue: arXiv cs.CR 2024
Year: 2024

There has been a large number of studies in interpretable and explainable ML for cybersecurity, in particular, for intrusion detection. Many of these studies have significant amount of overlapping and repeated evaluations and analysis. At the same time, these studies overlook crucial model, data, learning process, and utility related issues and many times completely disregard them. These issues include the use of overly complex and opaque ML models, unaccounted data imbalances and correlated features, inconsistent influential features across different explanation methods, the inconsistencies stemming from the constituents of a learning process, and the implausible utility of explanations. In this work, we empirically demonstrate these issues, analyze them and propose practical solutions in the context of feature-based model explanations. Specifically, we advise avoiding complex opaque models such as Deep Neural Networks and instead using interpretable ML models such as Decision Trees as the available intrusion datasets are not difficult for such interpretable models to classify successfully. Then, we bring attention to the binary classification metrics such as Matthews Correlation Coefficient (which are well-suited for imbalanced datasets. Moreover, we find that feature-based model explanations are most often inconsistent across different settings. In this respect, to further gauge the extent of inconsistencies, we introduce the notion of cross explanations which corroborates that the features that are determined to be impactful by one explanation method most often differ from those by another method. Furthermore, we show that strongly correlated data features and the constituents of a learning process, such as hyper-parameters and the optimization routine, become yet another source of inconsistent explanations. Finally, we discuss the utility of feature-based explanations.