OpenRaiser/Intern-Atlas huggingface dataset:
Improving Graph Convolutional Networks with Transformer Layer in social-based items recommendation
Venue: arXiv cs.LG 2024
Year: 2024

In this work, we have proposed an approach for improving the GCN for predicting ratings in social networks. Our model is expanded from the standard model with several layers of transformer architecture. The main focus of the paper is on the encoder architecture for node embedding in the network. Using the embedding layer from the graph-based convolution layer, the attention mechanism could rearrange the feature space to get a more efficient embedding for the downstream task. The experiments showed that our proposed architecture achieves better performance than GCN on the traditional link prediction task.
Sanity Checks Revisited: An Exploration to Repair the Model Parameter Randomisation Test
Venue: arXiv cs.AI 2024
Year: 2024

The Model Parameter Randomisation Test (MPRT) is widely acknowledged in the eXplainable Artificial Intelligence (XAI) community for its well-motivated evaluative principle: that the explanation function should be sensitive to changes in the parameters of the model function. However, recent works have identified several methodological caveats for the empirical interpretation of MPRT. To address these caveats, we introduce two adaptations to the original MPRT -- Smooth MPRT and Efficient MPRT, where the former minimises the impact that noise has on the evaluation results through sampling and the latter circumvents the need for biased similarity measurements by re-interpreting the test through the explanation's rise in complexity, after full parameter randomisation. Our experimental results demonstrate that these proposed variants lead to improved metric reliability, thus enabling a more trustworthy application of XAI methods.
Energy Patterns for Web: An Exploratory Study
Venue: arXiv cs.SE 2024
Year: 2024

As the energy footprint generated by software is increasing at an alarming rate, understanding how to develop energy-efficient applications has become a necessity. Previous work has introduced catalogs of coding practices, also known as energy patterns. These patterns are yet limited to Mobile or third-party libraries. In this study, we focus on the Web domain--a main source of energy consumption. First, we investigated whether and how Mobile energy patterns could be ported to this domain and found that 20 patterns could be ported. Then, we interviewed six expert web developers from different companies to challenge the ported patterns. Most developers expressed concerns for antipatterns, specifically with functional antipatterns, and were able to formulate guidelines to locate these patterns in the source code. Finally, to quantify the effect of Web energy patterns on energy consumption, we set up an automated pipeline to evaluate two ported patterns: 'Dynamic Retry Delay' (DRD) and 'Open Only When Necessary' (OOWN). With this, we found no evidence that the DRD pattern consumes less energy than its antipattern, while the opposite is true for OOWN. Data and Material: https://doi.org/10.5281/zenodo.8404487
Intention Analysis Makes LLMs A Good Jailbreak Defender
Venue: arXiv cs.CL 2024
Year: 2024

Aligning large language models (LLMs) with human values, particularly when facing complex and stealthy jailbreak attacks, presents a formidable challenge. Unfortunately, existing methods often overlook this intrinsic nature of jailbreaks, which limits their effectiveness in such complex scenarios. In this study, we present a simple yet highly effective defense strategy, i.e., Intention Analysis ($\mathbb{IA}$). $\mathbb{IA}$ works by triggering LLMs' inherent self-correct and improve ability through a two-stage process: 1) analyzing the essential intention of the user input, and 2) providing final policy-aligned responses based on the first round conversation. Notably, $\mathbb{IA}$ is an inference-only method, thus could enhance LLM safety without compromising their helpfulness. Extensive experiments on varying jailbreak benchmarks across a wide range of LLMs show that $\mathbb{IA}$ could consistently and significantly reduce the harmfulness in responses (averagely -48.2% attack success rate). Encouragingly, with our $\mathbb{IA}$, Vicuna-7B even outperforms GPT-3.5 regarding attack success rate. We empirically demonstrate that, to some extent, $\mathbb{IA}$ is robust to errors in generated intentions. Further analyses reveal the underlying principle of $\mathbb{IA}$: suppressing LLM's tendency to follow jailbreak prompts, thereby enhancing safety.
Cyborgs for strategic communication on social media
Venue: arXiv cs.SI 2024
Year: 2024

Social media platforms are a key ground of information consumption and dissemination. Key figures like politicians, celebrities and activists have leveraged on its wide user base for strategic communication. Strategic communications, or StratCom, is the deliberate act of information creation and distribution. Its techniques are used by these key figures for establishing their brand and amplifying their messages. Automated scripts are used on top of personal touches to quickly and effectively perform these tasks. The combination of automation and manual online posting creates a Cyborg social media profile, which is a hybrid between bot and human. In this study, we establish a quantitative definition for a Cyborg account, which is an account that are detected as bots in one time window, and identified as humans in another. This definition makes use of frequent changes of bot classification labels and large differences in bot likelihood scores to identify Cyborgs. We perform a large-scale analysis across over 3.1 million users from Twitter collected from two key events, the 2020 Coronavirus pandemic and 2020 US Elections. We extract Cyborgs from two datasets and employ tools from network science, natural language processing and manual annotation to characterize Cyborg accounts. Our analyses identify Cyborg accounts are mostly constructed for strategic communication uses, have a strong duality in their bot/human classification and are tactically positioned in the social media network, aiding these accounts to promote their desired content. Cyborgs are also discovered to have long online lives, indicating their ability to evade bot detectors, or the graciousness of platforms to allow their operations.
Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
Venue: arXiv cs.CL 2024
Year: 2024

Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision
OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favor of functional programming (FP), e.g., HumanEval and MBPP. To address this, our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k measures. Our evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights: 1) pass@o offers a more relevant and comprehensive assessment for OOP code generation; 2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT; 3) The poor performance of all advanced LLMs on our OOP benchmark highlights a critical need for improvements in this field. Our benchmark and scripts are publicly released at: https://github.com/alphadl/OOP-eval.
SeizNet: An AI-enabled Implantable Sensor Network System for Seizure Prediction
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we introduce SeizNet, a closed-loop system for predicting epileptic seizures through the use of Deep Learning (DL) method and implantable sensor networks. While pharmacological treatment is effective for some epilepsy patients (with ~65M people affected worldwide), one out of three suffer from drug-resistant epilepsy. To alleviate the impact of seizure, predictive systems have been developed that can notify such patients of an impending seizure, allowing them to take precautionary measures. SeizNet leverages DL techniques and combines data from multiple recordings, specifically intracranial electroencephalogram (iEEG) and electrocardiogram (ECG) sensors, that can significantly improve the specificity of seizure prediction while preserving very high levels of sensitivity. SeizNet DL algorithms are designed for efficient real-time execution at the edge, minimizing data privacy concerns, data transmission overhead, and power inefficiencies associated with cloud-based solutions. Our results indicate that SeizNet outperforms traditional single-modality and non-personalized prediction systems in all metrics, achieving up to 99% accuracy in predicting seizure, offering a promising new avenue in refractory epilepsy treatment.
Data-Efficient Interactive Multi-Objective Optimization Using ParEGO
Venue: arXiv cs.NE 2024
Year: 2024

Multi-objective optimization is a widely studied problem in diverse fields, such as engineering and finance, that seeks to identify a set of non-dominated solutions that provide optimal trade-offs among competing objectives. However, the computation of the entire Pareto front can become prohibitively expensive, both in terms of computational resources and time, particularly when dealing with a large number of objectives. In practical applications, decision-makers (DMs) will select a single solution of the Pareto front that aligns with their preferences to be implemented; thus, traditional multi-objective algorithms invest a lot of budget sampling solutions that are not interesting for the DM. In this paper, we propose two novel algorithms that employ Gaussian Processes and advanced discretization methods to efficiently locate the most preferred region of the Pareto front in expensive-to-evaluate problems. Our approach involves interacting with the decision-maker to guide the optimization process towards their preferred trade-offs. Our experimental results demonstrate that our proposed algorithms are effective in finding non-dominated solutions that align with the decision-maker's preferences while maintaining computational efficiency.
Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks
Venue: arXiv cs.CV 2024
Year: 2024

Feature removal is a central building block for eXplainable AI (XAI), both for occlusion-based explanations (Shapley values) as well as their evaluation (pixel flipping, PF). However, occlusion strategies can vary significantly from simple mean replacement up to inpainting with state-of-the-art diffusion models. This ambiguity limits the usefulness of occlusion-based approaches. For example, PF benchmarks lead to contradicting rankings. This is amplified by competing PF measures: Features are either removed starting with most influential first (MIF) or least influential first (LIF). This study proposes two complementary perspectives to resolve this disagreement problem. Firstly, we address the common criticism of occlusion-based XAI, that artificial samples lead to unreliable model evaluations. We propose to measure the reliability by the R(eference)-Out-of-Model-Scope (OMS) score. The R-OMS score enables a systematic comparison of occlusion strategies and resolves the disagreement problem by grouping consistent PF rankings. Secondly, we show that the insightfulness of MIF and LIF is conversely dependent on the R-OMS score. To leverage this, we combine the MIF and LIF measures into the symmetric relevance gain (SRG) measure. This breaks the inherent connection to the underlying occlusion strategy and leads to consistent rankings. This resolves the disagreement problem, which we verify for a set of 40 different occlusion strategies.
The SemIoE Ontology: A Semantic Model Solution for an IoE-based Industry
Venue: arXiv cs.IR 2024
Year: 2024

Recently, the Industry 5.0 is gaining attention as a novel paradigm, defining the next concrete steps toward more and more intelligent, green-aware and user-centric digital systems. In an era in which smart devices typically adopted in the industry domain are more and more sophisticated and autonomous, the Internet of Things and its evolution, known as the Internet of Everything (IoE, for short), involving also people, robots, processes and data in the network, represent the main driver to allow industries to put the experiences and needs of human beings at the center of their ecosystems. However, due to the extreme heterogeneity of the involved entities, their intrinsic need and capability to cooperate, and the aim to adapt to a dynamic user-centric context, special attention is required for the integration and processing of the data produced by such an IoE. This is the objective of the present paper, in which we propose a novel semantic model that formalizes the fundamental actors, elements and information of an IoE, along with their relationships. In our design, we focus on state-of-the-art design principles, in particular reuse, and abstraction, to build ``SemIoE'', a lightweight ontology inheriting and extending concepts from well-known and consolidated reference ontologies. The defined semantic layer represents a core data model that can be extended to embrace any modern industrial scenario. It represents the base of an IoE Knowledge Graph, on top of which, as an additional contribution, we analyze and define some essential services for an IoE-based industry.
User-Centric Cell-Free Wireless Networks for 6G: Communication Theoretic Models and Research Challenges
Venue: arXiv cs.IT 2024
Year: 2024

This paper presents a comprehensive communication theoretic model for the physical layer of a cell-free user-centric network, formed by user equipments (UEs), radio units (RUs), and decentralized units (DUs), uniformly spatially distributed over a given coverage area. We consider RUs equipped with multiple antennas, and focus on the regime where the UE, RU, and DU densities are constant and therefore the number of such nodes grows with the coverage area. A system is said scalable if the computing load and information rate at any node in the network converges to a constant as the network size (coverage area) grows to infinity. This imposes that each UE must be processed by a (user-centric) finite-size cluster of RUs, and that such cluster processors are dynamically allocated to the DUs (e.g., as software defined virtual network functions) in order to achieve a balanced computation load. We also assume that the RUs are connected to the DUs through a packet switching network, in order to achieve adaptive routing and load balance. For this model, we define in details the dynamic cluster formation and uplink pilot allocation. As a consequence of the pilot allocation and the scalability constraint, each cluster processor has a partial view of the network channel state information. We define the condition of ``ideal partial CSI'' when the channel vectors that can be estimated are perfectly known (while the ones that cannot be estimated are not know at all). We develop two attractive cluster-based linear receiver schemes for the uplink, and an uplink-downlink duality that allows to reuse such vectors as precoders for the downlink.
LLMRS: Unlocking Potentials of LLM-Based Recommender Systems for Software Purchase
Venue: arXiv cs.IR 2024
Year: 2024

Recommendation systems are ubiquitous, from Spotify playlist suggestions to Amazon product suggestions. Nevertheless, depending on the methodology or the dataset, these systems typically fail to capture user preferences and generate general recommendations. Recent advancements in Large Language Models (LLM) offer promising results for analyzing user queries. However, employing these models to capture user preferences and efficiency remains an open question. In this paper, we propose LLMRS, an LLM-based zero-shot recommender system where we employ pre-trained LLM to encode user reviews into a review score and generate user-tailored recommendations. We experimented with LLMRS on a real-world dataset, the Amazon product reviews, for software purchase use cases. The results show that LLMRS outperforms the ranking-based baseline model while successfully capturing meaningful information from product reviews, thereby providing more reliable recommendations.
Quantum Machine Learning in the Cognitive Domain: Alzheimer's Disease Study
Venue: arXiv cs.LG 2024
Year: 2024

Alzheimer's disease (AD) is the most prevalent neurodegenerative disorder, primarily affecting the elderly population and leading to significant cognitive decline. This decline manifests in various mental faculties such as attention, memory, and higher-order cognitive functions, severely impacting an individual's ability to comprehend information, acquire new knowledge, and communicate effectively. One of the tasks influenced by cognitive impairments is handwriting. By analyzing specific features of handwriting, including pressure, velocity, and spatial organization, researchers can detect subtle changes that may indicate early-stage cognitive impairments, particularly AD. Recent developments in classical artificial intelligence (AI) methods have shown promise in detecting AD through handwriting analysis. However, as the dataset size increases, these AI approaches demand greater computational resources, and diagnoses are often affected by limited classical vector spaces and feature correlations. Recent studies have shown that quantum computing technologies, developed by harnessing the unique properties of quantum particles such as superposition and entanglement, can not only address the aforementioned problems but also accelerate complex data analysis and enable more efficient processing of large datasets. In this study, we propose a variational quantum classifier with fewer circuit elements to facilitate early AD diagnosis based on handwriting data. Our model has demonstrated comparable classification performance to classical methods and underscores the potential of quantum computing models in addressing cognitive problems, paving the way for future research in this domain.
Reliability Analysis of Psychological Concept Extraction and Classification in User-penned Text
Venue: arXiv cs.CL 2024
Year: 2024

The social NLP research community witness a recent surge in the computational advancements of mental health analysis to build responsible AI models for a complex interplay between language use and self-perception. Such responsible AI models aid in quantifying the psychological concepts from user-penned texts on social media. On thinking beyond the low-level (classification) task, we advance the existing binary classification dataset, towards a higher-level task of reliability analysis through the lens of explanations, posing it as one of the safety measures. We annotate the LoST dataset to capture nuanced textual cues that suggest the presence of low self-esteem in the posts of Reddit users. We further state that the NLP models developed for determining the presence of low self-esteem, focus more on three types of textual cues: (i) Trigger: words that triggers mental disturbance, (ii) LoST indicators: text indicators emphasizing low self-esteem, and (iii) Consequences: words describing the consequences of mental disturbance. We implement existing classifiers to examine the attention mechanism in pre-trained language models (PLMs) for a domain-specific psychology-grounded task. Our findings suggest the need of shifting the focus of PLMs from Trigger and Consequences to a more comprehensive explanation, emphasizing LoST indicators while determining low self-esteem in Reddit posts.
Synthetic Data Generation Framework, Dataset, and Efficient Deep Model for Pedestrian Intention Prediction
Venue: arXiv cs.CV 2024
Year: 2024

Pedestrian intention prediction is crucial for autonomous driving. In particular, knowing if pedestrians are going to cross in front of the ego-vehicle is core to performing safe and comfortable maneuvers. Creating accurate and fast models that predict such intentions from sequential images is challenging. A factor contributing to this is the lack of datasets with diverse crossing and non-crossing (C/NC) scenarios. We address this scarceness by introducing a framework, named ARCANE, which allows programmatically generating synthetic datasets consisting of C/NC video clip samples. As an example, we use ARCANE to generate a large and diverse dataset named PedSynth. We will show how PedSynth complements widely used real-world datasets such as JAAD and PIE, so enabling more accurate models for C/NC prediction. Considering the onboard deployment of C/NC prediction models, we also propose a deep model named PedGNN, which is fast and has a very low memory footprint. PedGNN is based on a GNN-GRU architecture that takes a sequence of pedestrian skeletons as input to predict crossing intentions.
Seeing the roads through the trees: A benchmark for modeling spatial dependencies with aerial imagery
Venue: arXiv cs.CV 2024
Year: 2024

Fully understanding a complex high-resolution satellite or aerial imagery scene often requires spatial reasoning over a broad relevant context. The human object recognition system is able to understand object in a scene over a long-range relevant context. For example, if a human observes an aerial scene that shows sections of road broken up by tree canopy, then they will be unlikely to conclude that the road has actually been broken up into disjoint pieces by trees and instead think that the canopy of nearby trees is occluding the road. However, there is limited research being conducted to understand long-range context understanding of modern machine learning models. In this work we propose a road segmentation benchmark dataset, Chesapeake Roads Spatial Context (RSC), for evaluating the spatial long-range context understanding of geospatial machine learning models and show how commonly used semantic segmentation models can fail at this task. For example, we show that a U-Net trained to segment roads from background in aerial imagery achieves an 84% recall on unoccluded roads, but just 63.5% recall on roads covered by tree canopy despite being trained to model both the same way. We further analyze how the performance of models changes as the relevant context for a decision (unoccluded roads in our case) varies in distance. We release the code to reproduce our experiments and dataset of imagery and masks to encourage future research in this direction -- https://github.com/isaaccorley/ChesapeakeRSC.