OpenRaiser/Intern-Atlas huggingface dataset:
GreenWhisk: Emission-Aware Computing for Serverless Platform
Venue: arXiv cs.DC 2024
Year: 2024

Serverless computing is an emerging cloud computing abstraction wherein the cloud platform transparently manages all resources, including explicitly provisioning resources and geographical load balancing when the demand for service spikes. Users provide code as functions, and the cloud platform runs these functions handling all aspects of function execution. While prior work has primarily focused on optimizing performance, this paper focuses on reducing the carbon footprint of these systems making variations in grid carbon intensity and intermittency from renewables transparent to the user. We introduce GreenWhisk, a carbon-aware serverless computing platform built upon Apache OpenWhisk, operating in two modes - grid-connected and grid-isolated - addressing intermittency challenges arising from renewables and the grid's carbon footprint. Moreover, we develop carbon-aware load balancing algorithms that leverage energy and carbon information to reduce the carbon footprint. Our evaluation results show that GreenWhisk can easily incorporate carbon-aware algorithms, thereby reducing the carbon footprint of functions without significantly impacting the performance of function execution. In doing so, our system design enables the integration of new carbon-aware strategies into a serverless computing platform.
VECA: Reliable and Confidential Resource Clustering for Volunteer Edge-Cloud Computing
Venue: arXiv cs.NI 2024
Year: 2024

Volunteer Edge-Cloud (VEC) computing has a significant potential to support scientific workflows in user communities contributing volunteer edge nodes. However, managing heterogeneous and intermittent resources to support machine/deep learning (ML/DL) based workflows poses challenges in resource governance for reliability, and confidentiality for model/data privacy protection. There is a need for approaches to handle the volatility of volunteer edge node availability, and also to scale the confidential data-intensive workflow execution across a large number of VEC nodes. In this paper, we present VECA, a reliable and confidential VEC resource clustering solution featuring three-fold methods tailored for executing ML/DL-based scientific workflows on VEC resources. Firstly, a capacity-based clustering approach enhances system reliability and minimizes VEC node search latency. Secondly, a novel two-phase, globally distributed scheduling scheme optimizes job allocation based on node attributes and using time-series-based Recurrent Neural Networks. Lastly, the integration of confidential computing ensures privacy preservation of the scientific workflows, where model and data information are not shared with VEC resources providers. We evaluate VECA in a Function-as-a-Service (FaaS) cloud testbed that features OpenFaaS and MicroK8S to support two ML/DL-based scientific workflows viz., G2P-Deep (bioinformatics) and PAS-ML (health informatics). Results from tested experiments demonstrate that our proposed VECA approach outperforms state-of-the-art methods; especially VECA exhibits a two-fold reduction in VEC node search latency and over 20% improvement in productivity rates following execution failures compared to the next best method.
A Comparative Study of Offline Models and Online LLMs in Fake News Detection
Venue: arXiv cs.SI 2024
Year: 2024

Fake news detection remains a critical challenge in today's rapidly evolving digital landscape, where misinformation can spread faster than ever before. Traditional fake news detection models often rely on static datasets and auxiliary information, such as metadata or social media interactions, which limits their adaptability to real-time scenarios. Recent advancements in Large Language Models (LLMs) have demonstrated significant potential in addressing these challenges due to their extensive pre-trained knowledge and ability to analyze textual content without relying on auxiliary data. However, many of these LLM-based approaches are still rooted in static datasets, with limited exploration into their real-time processing capabilities. This paper presents a systematic evaluation of both traditional offline models and state-of-the-art LLMs for real-time fake news detection. We demonstrate the limitations of existing offline models, including their inability to adapt to dynamic misinformation patterns. Furthermore, we show that newer LLM models with online capabilities, such as GPT-4, Claude, and Gemini, are better suited for detecting emerging fake news in real-time contexts. Our findings emphasize the importance of transitioning from offline to online LLM models for real-time fake news detection. Additionally, the public accessibility of LLMs enhances their scalability and democratizes the tools needed to combat misinformation. By leveraging real-time data, our work marks a significant step toward more adaptive, effective, and scalable fake news detection systems.
Generative Manufacturing: A requirements and resource-driven approach to part making
Venue: arXiv cs.CE 2024
Year: 2024

Advances in CAD and CAM have enabled engineers and design teams to digitally design parts with unprecedented ease. Software solutions now come with a range of modules for optimizing designs for performance requirements, generating instructions for manufacturing, and digitally tracking the entire process from design to procurement in the form of product life-cycle management tools. However, existing solutions force design teams and corporations to take a primarily serial approach where manufacturing and procurement decisions are largely contingent on design, rather than being an integral part of the design process. In this work, we propose a new approach to part making where design, manufacturing, and supply chain requirements and resources can be jointly considered and optimized. We present the Generative Manufacturing compiler that accepts as input the following: 1) An engineering part requirements specification that includes quantities such as loads, domain envelope, mass, and compliance, 2) A business part requirements specification that includes production volume, cost, and lead time, 3) Contextual knowledge about the current manufacturing state such as availability of relevant manufacturing equipment, materials, and workforce, both locally and through the supply chain. Based on these factors, the compiler generates and evaluates manufacturing process alternatives and the optimal derivative designs that are implied by each process, and enables a user guided iterative exploration of the design space. As part of our initial implementation of this compiler, we demonstrate the effectiveness of our approach on examples of a cantilever beam problem and a rocket engine mount problem and showcase its utility in creating and selecting optimal solutions according to the requirements and resources.
FIDAVL: Fake Image Detection and Attribution using Vision-Language Model
Venue: arXiv cs.CV 2024
Year: 2024

We introduce FIDAVL: Fake Image Detection and Attribution using a Vision-Language Model. FIDAVL is a novel and efficient mul-titask approach inspired by the synergies between vision and language processing. Leveraging the benefits of zero-shot learning, FIDAVL exploits the complementarity between vision and language along with soft prompt-tuning strategy to detect fake images and accurately attribute them to their originating source models. We conducted extensive experiments on a comprehensive dataset comprising synthetic images generated by various state-of-the-art models. Our results demonstrate that FIDAVL achieves an encouraging average detection accuracy of 95.42% and F1-score of 95.47% while also obtaining noteworthy performance metrics, with an average F1-score of 92.64% and ROUGE-L score of 96.50% for attributing synthetic images to their respective source generation models. The source code of this work will be publicly released at https://github.com/Mamadou-Keita/FIDAVL.
Well, that escalated quickly: The Single-Turn Crescendo Attack (STCA)
Venue: arXiv cs.CR 2024
Year: 2024

This paper introduces a new method for adversarial attacks on large language models (LLMs) called the Single-Turn Crescendo Attack (STCA). Building on the multi-turn crescendo attack method introduced by Russinovich, Salem, and Eldan (2024), which gradually escalates the context to provoke harmful responses, the STCA achieves similar outcomes in a single interaction. By condensing the escalation into a single, well-crafted prompt, the STCA bypasses typical moderation filters that LLMs use to prevent inappropriate outputs. This technique reveals vulnerabilities in current LLMs and emphasizes the importance of stronger safeguards in responsible AI (RAI). The STCA offers a novel method that has not been previously explored.
Discovering Cyclists' Visual Preferences Through Shared Bike Trajectories and Street View Images Using Inverse Reinforcement Learning
Venue: arXiv cs.LG 2024
Year: 2024

Cycling has gained global popularity for its health benefits and positive urban impacts. To effectively promote cycling, early studies have extensively investigated the relationship between cycling behaviors and environmental factors, especially cyclists' preferences when making route decisions. However, these studies often struggle to comprehensively describe detailed cycling procedures at a large scale due to data limitations, and they tend to overlook the complex nature of cyclists' preferences. To address these issues, we propose a novel framework aimed to quantify and interpret cyclists' complicated visual preferences by leveraging maximum entropy deep inverse reinforcement learning(MEDIRL)and explainable artificial intelligence(XAI). Implemented in Bantian Sub-district, Shenzhen, we adapt MEDIRL model for efficient estimation of cycling reward function by integrating dockless-bike-sharing(DBS) trajectory and street view images(SVIs), which serves as a representation of cyclists' preferences for street visual environments during routing. In addition, we demonstrate the feasibility and reliability of MEDIRL in discovering cyclists' visual preferences. We find that cyclists focus on specific street visual elements when making route decisions, which can be summarized as their attention to safety, street enclosure, and cycling comfort. Further analysis reveals the complex nonlinear effects of street visual elements on cyclists' preferences, offering a cost-effective perspective on streetscapes design. Our proposed framework advances the understanding of individual cycling behaviors and provides actionable insights for urban planners to design bicycle-friendly streetscapes that prioritize cyclists' preferences.
Debate on Graph: a Flexible and Reliable Reasoning Framework for Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) may suffer from hallucinations in real-world applications due to the lack of relevant knowledge. In contrast, knowledge graphs encompass extensive, multi-relational structures that store a vast array of symbolic facts. Consequently, integrating LLMs with knowledge graphs has been extensively explored, with Knowledge Graph Question Answering (KGQA) serving as a critical touchstone for the integration. This task requires LLMs to answer natural language questions by retrieving relevant triples from knowledge graphs. However, existing methods face two significant challenges: \textit{excessively long reasoning paths distracting from the answer generation}, and \textit{false-positive relations hindering the path refinement}. In this paper, we propose an iterative interactive KGQA framework that leverages the interactive learning capabilities of LLMs to perform reasoning and Debating over Graphs (DoG). Specifically, DoG employs a subgraph-focusing mechanism, allowing LLMs to perform answer trying after each reasoning step, thereby mitigating the impact of lengthy reasoning paths. On the other hand, DoG utilizes a multi-role debate team to gradually simplify complex questions, reducing the influence of false-positive relations. This debate mechanism ensures the reliability of the reasoning process. Experimental results on five public datasets demonstrate the effectiveness and superiority of our architecture. Notably, DoG outperforms the state-of-the-art method ToG by 23.7\% and 9.1\% in accuracy on WebQuestions and GrailQA, respectively. Furthermore, the integration experiments with various LLMs on the mentioned datasets highlight the flexibility of DoG. Code is available at \url{https://github.com/reml-group/DoG}.
InfraLib: Enabling Reinforcement Learning and Decision-Making for Large-Scale Infrastructure Management
Venue: arXiv cs.AI 2024
Year: 2024

Efficient management of infrastructure systems is crucial for economic stability, sustainability, and public safety. However, infrastructure sustainment is challenging due to the vast scale of systems, stochastic deterioration of components, partial observability, and resource constraints. Decision-making strategies that rely solely on human judgment often result in suboptimal decisions over large scales and long horizons. While data-driven approaches like reinforcement learning offer promising solutions, their application has been limited by the lack of suitable simulation environments. We present InfraLib, an open-source modular and extensible framework that enables modeling and analyzing infrastructure management problems with resource constraints as sequential decision-making problems. The framework implements hierarchical, stochastic deterioration models, supports realistic partial observability, and handles practical constraints including cyclical budgets and component unavailability. InfraLib provides standardized environments for benchmarking decision-making approaches, along with tools for expert data collection and policy evaluation. Through case studies on both synthetic benchmarks and real-world road networks, we demonstrate InfraLib's ability to model diverse infrastructure management scenarios while maintaining computational efficiency at scale.
Two or three things I know about tree transducers
Venue: arXiv cs.FL 2024
Year: 2024

You might know that the name "tree transducers" refers to various kinds of automata that compute functions on ranked trees, i.e. terms over a first-order signature. But have you ever wondered about how to remember what a macro tree transducer does? Or what are the connections between top-down tree(-to-string) transducers, multi bottom-up tree(-to-string) transducers, tree-walking transducers, (invisible) pebble tree transducers, monadic second-order transductions, unfoldings of rooted directed acyclic graphs (i.e. term graphs) -- and what happens when the functions that they compute are composed? The answers may be found in old papers (mostly coauthored by Engelfriet), but maybe you can save some time by first looking at this short note.
Active Fake: DeepFake Camouflage
Venue: arXiv cs.CV 2024
Year: 2024

DeepFake technology has gained significant attention due to its ability to manipulate facial attributes with high realism, raising serious societal concerns. Face-Swap DeepFake is the most harmful among these techniques, which fabricates behaviors by swapping original faces with synthesized ones. Existing forensic methods, primarily based on Deep Neural Networks (DNNs), effectively expose these manipulations and have become important authenticity indicators. However, these methods mainly concentrate on capturing the blending inconsistency in DeepFake faces, raising a new security issue, termed Active Fake, emerges when individuals intentionally create blending inconsistency in their authentic videos to evade responsibility. This tactic is called DeepFake Camouflage. To achieve this, we introduce a new framework for creating DeepFake camouflage that generates blending inconsistencies while ensuring imperceptibility, effectiveness, and transferability. This framework, optimized via an adversarial learning strategy, crafts imperceptible yet effective inconsistencies to mislead forensic detectors. Extensive experiments demonstrate the effectiveness and robustness of our method, highlighting the need for further research in active fake detection.
TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations
Venue: arXiv cs.CV 2024
Year: 2024

Multimodal Large Language Models (MLLMs) have significantly improved performance across various image-language applications. Recently, there has been a growing interest in adapting image pre-trained MLLMs for video-related tasks. However, most efforts concentrate on enhancing the vision encoder and projector components, while the core part, Large Language Models (LLMs), remains comparatively under-explored. In this paper, we propose two strategies to enhance the model's capability in video understanding tasks by improving inter-layer attention computation in LLMs. Specifically, the first approach focuses on the enhancement of Rotary Position Embedding (RoPE) with Temporal-Aware Dual RoPE, which introduces temporal position information to strengthen the MLLM's temporal modeling capabilities while preserving the relative position relationships of both visual and text tokens. The second approach involves enhancing the Attention Mask with the Frame-wise Block Causal Attention Mask, a simple yet effective method that broadens visual token interactions within and across video frames while maintaining the causal inference mechanism. Based on these proposed methods, we adapt LLaVA for video understanding tasks, naming it Temporal-Considered LLaVA (TC-LLaVA). Our TC-LLaVA achieves new state-of-the-art performance across various video understanding benchmarks with only supervised fine-tuning (SFT) on video-related datasets.
Application Research On Real-Time Perception Of Device Performance Status
Venue: arXiv cs.PF 2024
Year: 2024

In order to accurately identify the performance status of mobile devices and finely adjust the user experience, a real-time performance perception evaluation method based on TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) combined with entropy weighting method and time series model construction was studied. After collecting the performance characteristics of various mobile devices, the device performance profile was fitted by using PCA (principal component analysis) dimensionality reduction and feature engineering methods such as descriptive time series analysis. The ability of performance features and profiles to describe the real-time performance status of devices was understood and studied by applying the TOPSIS method and multi-level weighting processing. A time series model was constructed for the feature set under objective weighting, and multiple sensitivity (real-time, short-term, long-term) performance status perception results were provided to obtain real-time performance evaluation data and long-term stable performance prediction data. Finally, by configuring dynamic AB experiments and overlaying fine-grained power reduction strategies, the usability of the method was verified, and the accuracy of device performance status identification and prediction was compared with the performance of the profile features including dimensionality reduction time series modeling, TOPSIS method and entropy weighting method, subjective weighting, HMA method. The results show that accurate real-time performance perception results can greatly enhance business value, and this research has application effectiveness and certain forward-looking significance.
FairQuant: Certifying and Quantifying Fairness of Deep Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

We propose a method for formally certifying and quantifying individual fairness of deep neural networks (DNN). Individual fairness guarantees that any two individuals who are identical except for a legally protected attribute (e.g., gender or race) receive the same treatment. While there are existing techniques that provide such a guarantee, they tend to suffer from lack of scalability or accuracy as the size and input dimension of the DNN increase. Our method overcomes this limitation by applying abstraction to a symbolic interval based analysis of the DNN followed by iterative refinement guided by the fairness property. Furthermore, our method lifts the symbolic interval based analysis from conventional qualitative certification to quantitative certification, by computing the percentage of individuals whose classification outputs are provably fair, instead of merely deciding if the DNN is fair. We have implemented our method and evaluated it on deep neural networks trained on four popular fairness research datasets. The experimental results show that our method is not only more accurate than state-of-the-art techniques but also several orders-of-magnitude faster.
Dual-TSST: A Dual-Branch Temporal-Spectral-Spatial Transformer Model for EEG Decoding
Venue: arXiv cs.HC 2024
Year: 2024

The decoding of electroencephalography (EEG) signals allows access to user intentions conveniently, which plays an important role in the fields of human-machine interaction. To effectively extract sufficient characteristics of the multichannel EEG, a novel decoding architecture network with a dual-branch temporal-spectral-spatial transformer (Dual-TSST) is proposed in this study. Specifically, by utilizing convolutional neural networks (CNNs) on different branches, the proposed processing network first extracts the temporal-spatial features of the original EEG and the temporal-spectral-spatial features of time-frequency domain data converted by wavelet transformation, respectively. These perceived features are then integrated by a feature fusion block, serving as the input of the transformer to capture the global long-range dependencies entailed in the non-stationary EEG, and being classified via the global average pooling and multi-layer perceptron blocks. To evaluate the efficacy of the proposed approach, the competitive experiments are conducted on three publicly available datasets of BCI IV 2a, BCI IV 2b, and SEED, with the head-to-head comparison of more than ten other state-of-the-art methods. As a result, our proposed Dual-TSST performs superiorly in various tasks, which achieves the promising EEG classification performance of average accuracy of 80.67% in BCI IV 2a, 88.64% in BCI IV 2b, and 96.65% in SEED, respectively. Extensive ablation experiments conducted between the Dual-TSST and comparative baseline model also reveal the enhanced decoding performance with each module of our proposed method. This study provides a new approach to high-performance EEG decoding, and has great potential for future CNN-Transformer based applications.
E2CL: Exploration-based Error Correction Learning for Embodied Agents
Venue: arXiv cs.CL 2024
Year: 2024

Language models are exhibiting increasing capability in knowledge utilization and reasoning. However, when applied as agents in embodied environments, they often suffer from misalignment between their intrinsic knowledge and environmental knowledge, leading to infeasible actions. Traditional environment alignment methods, such as supervised learning on expert trajectories and reinforcement learning, encounter limitations in covering environmental knowledge and achieving efficient convergence, respectively. Inspired by human learning, we propose Exploration-based Error Correction Learning (E2CL), a novel framework that leverages exploration-induced errors and environmental feedback to enhance environment alignment for embodied agents. E2CL incorporates teacher-guided and teacher-free explorations to gather environmental feedback and correct erroneous actions. The agent learns to provide feedback and self-correct, thereby enhancing its adaptability to target environments. Extensive experiments in the VirtualHome environment demonstrate that E2CL-trained agents outperform those trained by baseline methods and exhibit superior self-correction capabilities.