OpenRaiser/Intern-Atlas huggingface dataset:
Alea-BFT: Practical Asynchronous Byzantine Fault Tolerance
Venue: arXiv cs.DC 2024
Year: 2024

Traditional Byzantine Fault Tolerance (BFT) state machine replication protocols assume a partial synchrony model, leading to a design where a leader replica drives the protocol and is replaced after a timeout. Recently, we witnessed a surge of asynchronous BFT protocols, which use randomization to remove the need for bounds on message delivery times, making them more resilient to adverse network conditions. However, existing research proposals still fall short of gaining practical adoption, plausibly because they are not able to combine good performance with a simple design that can be readily understood and adopted. In this paper, we present Alea-BFT, a simple and highly efficient asynchronous BFT protocol, which is gaining practical adoption, namely in Ethereum distributed validators. Alea-BFT brings the key design insight from classical protocols of concentrating part of the work on a single designated replica and incorporates this principle in a simple two-stage pipelined design, with an efficient broadcast led by the designated replica, followed by an inexpensive binary agreement. The evaluation of our research prototype implementation and two real-world integrations in cryptocurrency ecosystems shows excellent performance, improving on the fastest protocol (Dumbo-NG) in terms of latency and displaying good performance under faults.
A Novel Skiagraphic Method of Casting Shade of a Torus
Venue: arXiv cs.GR 2024
Year: 2024

This paper introduces a novel skiagraphic method for shading toroidal forms in architectural illustrations, addressing the challenges of traditional techniques. Skiagraphy projects 3D objects onto 2D surfaces to display geometric properties. Traditional shading of tori involves extensive manual calculations and multiple projections, leading to high complexity and inaccuracies. The proposed method simplifies this by focusing on the elevation view, eliminating the need for multiple projections and complex math. Utilizing descriptive geometry, it reduces labor and complexity. Accuracy was validated through comparisons with SketchUp-generated shading and various torus configurations. This technique streamlines shading toroidal shapes while maintaining the artistic value of traditional illustration. Additionally, it has potential applications in 3D model generation from architectural shade casts, contributing to the evolving field of architectural visualization and representation.
Metasurface Energy Harvesters: State-of-the-Art Designs and Their Potential for Energy Sustainable Reconfigurable Intelligent Surfaces
Venue: arXiv cs.ET 2024
Year: 2024

Metasurface Energy Harvesters (MEHs) have emerged as a prominent enabler of highly efficient Radio Frequency (RF) energy harvesters. This survey delves into the fundamentals of the MEH technology, providing a comprehensive overview of their working principle, unit cell designs and prototypes over various frequency bands, as well as state-of-the art modes of operation. Inspired by the recent academic and industrial interest on Reconfigurable Intelligent Surfaces (RISs)for the upcoming sixth-Generation (6G) of wireless networks, we study the interplay between this technology and MEHs aiming for energy sustainable RISs power by metasurface-based RF energy harvesting. We present a novel hybrid unit cell design capable of simultaneous energy harvesting and 1-bit tunable reflection whose dual-functional response is validated via full-wave simulations. Then, we conduct a comparative collection of real-world measurements for ambient RF power levels and power consumption budgets of reflective RISs to unveil the potential for a self-sustainable RIS via ambient RF energy harvesting. The paper is concluded with an elaborative discussion on open design challenges and future research directions for MEHs and energy sustainable hybrid RISs.
Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context
Venue: arXiv cs.CL 2024
Year: 2024

Previous research on testing the vulnerabilities in Large Language Models (LLMs) using adversarial attacks has primarily focused on nonsensical prompt injections, which are easily detected upon manual or automated review (e.g., via byte entropy). However, the exploration of innocuous human-understandable malicious prompts augmented with adversarial injections remains limited. In this research, we explore converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. This allows us to show suffix conversion without any gradients, using only LLMs to perform the attacks, and thus better understand the scope of possible risks. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. The situations are extracted from the IMDB dataset, and prompts are defined following a few-shot chain-of-thought prompting. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs. We find that across many LLMs, as few as 1 attempt produces an attack and that these attacks transfer between LLMs.
Universally Harmonizing Differential Privacy Mechanisms for Federated Learning: Boosting Accuracy and Convergence
Venue: arXiv cs.LG 2024
Year: 2024

Differentially private federated learning (DP-FL) is a promising technique for collaborative model training while ensuring provable privacy for clients. However, optimizing the tradeoff between privacy and accuracy remains a critical challenge. To our best knowledge, we propose the first DP-FL framework (namely UDP-FL), which universally harmonizes any randomization mechanism (e.g., an optimal one) with the Gaussian Moments Accountant (viz. DP-SGD) to significantly boost accuracy and convergence. Specifically, UDP-FL demonstrates enhanced model performance by mitigating the reliance on Gaussian noise. The key mediator variable in this transformation is the Rényi Differential Privacy notion, which is carefully used to harmonize privacy budgets. We also propose an innovative method to theoretically analyze the convergence for DP-FL (including our UDP-FL ) based on mode connectivity analysis. Moreover, we evaluate our UDP-FL through extensive experiments benchmarked against state-of-the-art (SOTA) methods, demonstrating superior performance on both privacy guarantees and model performance. Notably, UDP-FL exhibits substantial resilience against different inference attacks, indicating a significant advance in safeguarding sensitive data in federated learning environments.
Downstream-Pretext Domain Knowledge Traceback for Active Learning
Venue: arXiv cs.LG 2024
Year: 2024

Active learning (AL) is designed to construct a high-quality labeled dataset by iteratively selecting the most informative samples. Such sampling heavily relies on data representation, while recently pre-training is popular for robust feature learning. However, as pre-training utilizes low-level pretext tasks that lack annotation, directly using pre-trained representation in AL is inadequate for determining the sampling score. To address this problem, we propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance for selecting diverse and instructive samples near the decision boundary. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. The diversity indicator constructs two feature spaces based on the pre-training pretext model and the downstream knowledge from annotation, by which it locates the neighbors of unlabeled data from the downstream space in the pretext space to explore the interaction of samples. With this mechanism, DOKT unifies the data relations of low-level and high-level representations to estimate traceback diversity. Next, in the uncertainty estimator, domain mixing is designed to enforce perceptual perturbing to unlabeled samples with similar visual patches in the pretext space. Then the divergence of perturbed samples is measured to estimate the domain uncertainty. As a result, DOKT selects the most diverse and important samples based on these two modules. The experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods and generalizes well to various application scenarios such as semantic segmentation and image captioning.
CrowdMAC: Masked Crowd Density Completion for Robust Crowd Density Forecasting
Venue: arXiv cs.CV 2024
Year: 2024

A crowd density forecasting task aims to predict how the crowd density map will change in the future from observed past crowd density maps. However, the past crowd density maps are often incomplete due to the miss-detection of pedestrians, and it is crucial to develop a robust crowd density forecasting model against the miss-detection. This paper presents a MAsked crowd density Completion framework for crowd density forecasting (CrowdMAC), which is simultaneously trained to forecast future crowd density maps from partially masked past crowd density maps (i.e., forecasting maps from past maps with miss-detection) while reconstructing the masked observation maps (i.e., imputing past maps with miss-detection). Additionally, we propose Temporal-Density-aware Masking (TDM), which non-uniformly masks tokens in the observed crowd density map, considering the sparsity of the crowd density maps and the informativeness of the subsequent frames for the forecasting task. Moreover, we introduce multi-task masking to enhance training efficiency. In the experiments, CrowdMAC achieves state-of-the-art performance on seven large-scale datasets, including SDD, ETH-UCY, inD, JRDB, VSCrowd, FDST, and croHD. We also demonstrate the robustness of the proposed method against both synthetic and realistic miss-detections. The code is released at https://fujiry0.github.io/CrowdMAC-project-page.
Understanding the Needs of Nonhuman Stakeholders in Design Process: An Overview of and Reflection on Methods
Venue: arXiv cs.HC 2024
Year: 2024

Design practice traditionally focused on human concerns, either overseeing the various effects of climate issues on nonhuman stakeholders or considering them as resources to address these problems. The climate crisis's urgency demands a design shift towards sustainability and inclusivity. This shift was happening through an emerging theme in design, More-Than-Human (MTH), which expands the notion of the user to animals, things, nature, and microbes. Such an expansion creates a requirement for designers to consider nonhuman perspectives during the design process. This paper investigates the methods used in MTH Design studies to explore and synthesize the perspectives of nonhuman users. Reviewing 30 papers, it highlights a predominant focus on animals and things over plants and microbes in MTH studies, along with a scarcity of synthesis methods. It identifies the necessity of tools that represent nonhumans with their relationships within larger ecosystems, and calls for increased attention to plants and microbes, emphasizing their vital role in sustainable environments and urging researchers to develop methods for understanding these species. By highlighting method strengths and weaknesses, it aims to guide designers and design researchers who plan to work with nonhuman users in selecting appropriate methods.
Enhancing Skin Disease Classification Leveraging Transformer-based Deep Learning Architectures and Explainable AI
Venue: arXiv cs.CV 2024
Year: 2024

Skin diseases affect over a third of the global population, yet their impact is often underestimated. Automating skin disease classification to assist doctors with their prognosis might be difficult. Nevertheless, due to efficient feature extraction pipelines, deep learning techniques have shown much promise for various tasks, including dermatological disease identification. This study uses a skin disease dataset with 31 classes and compares it with all versions of Vision Transformers, Swin Transformers and DivoV2. The analysis is also extended to compare with benchmark convolution-based architecture presented in the literature. Transfer learning with ImageNet1k weights on the skin disease dataset contributes to a high test accuracy of 96.48\% and an F1-Score of 0.9727 using DinoV2, which is almost a 10\% improvement over this data's current benchmark results. The performance of DinoV2 was also compared for the HAM10000 and Dermnet datasets to test the model's robustness, and the trained model overcomes the benchmark results by a slight margin in test accuracy and in F1-Score on the 23 and 7 class datasets. The results are substantiated using explainable AI frameworks like GradCAM and SHAP, which provide precise image locations to map the disease, assisting dermatologists in early detection, prompt prognosis, and treatment.
A Two-Phase Visualization System for Continuous Human-AI Collaboration in Sequelae Analysis and Modeling
Venue: arXiv cs.HC 2024
Year: 2024

In healthcare, AI techniques are widely used for tasks like risk assessment and anomaly detection. Despite AI's potential as a valuable assistant, its role in complex medical data analysis often oversimplifies human-AI collaboration dynamics. To address this, we collaborated with a local hospital, engaging six physicians and one data scientist in a formative study. From this collaboration, we propose a framework integrating two-phase interactive visualization systems: one for Human-Led, AI-Assisted Retrospective Analysis and another for AI-Mediated, Human-Reviewed Iterative Modeling. This framework aims to enhance understanding and discussion around effective human-AI collaboration in healthcare.
VisFly: An Efficient and Versatile Simulator for Training Vision-based Flight
Venue: arXiv cs.RO 2024
Year: 2024

We present VisFly, a quadrotor simulator designed to efficiently train vision-based flight policies using reinforcement learning algorithms. VisFly offers a user-friendly framework and interfaces, leveraging Habitat-Sim's rendering engines to achieve frame rates exceeding 10,000 frames per second for rendering motion and sensor data. The simulator incorporates differentiable physics and is seamlessly wrapped with the Gym environment, facilitating the straightforward implementation of various learning algorithms. It supports the directly importing open-source scene datasets compatible with Habitat-Sim, enabling training on diverse real-world environments simultaneously. To validate our simulator, we also make three reinforcement learning examples for typical flight tasks relying on visual observations. The simulator is now available at [https://github.com/SJTU-ViSYS-team/VisFly].
FedPartWhole: Federated domain generalization via consistent part-whole hierarchies
Venue: arXiv cs.CV 2024
Year: 2024

Federated Domain Generalization (FedDG), aims to tackle the challenge of generalizing to unseen domains at test time while catering to the data privacy constraints that prevent centralized data storage from different domains originating at various clients. Existing approaches can be broadly categorized into four groups: domain alignment, data manipulation, learning strategies, and optimization of model aggregation weights. This paper proposes a novel approach to Federated Domain Generalization that tackles the problem from the perspective of the backbone model architecture. The core principle is that objects, even under substantial domain shifts and appearance variations, maintain a consistent hierarchical structure of parts and wholes. For instance, a photograph and a sketch of a dog share the same hierarchical organization, consisting of a head, body, limbs, and so on. The introduced architecture explicitly incorporates a feature representation for the image parse tree. To the best of our knowledge, this is the first work to tackle Federated Domain Generalization from a model architecture standpoint. Our approach outperforms a convolutional architecture of comparable size by over 12\%, despite utilizing fewer parameters. Additionally, it is inherently interpretable, contrary to the black-box nature of CNNs, which fosters trust in its predictions, a crucial asset in federated learning.
Automatic Real-word Error Correction in Persian Text
Venue: arXiv cs.CL 2024
Year: 2024

Automatic spelling correction stands as a pivotal challenge within the ambit of natural language processing (NLP), demanding nuanced solutions. Traditional spelling correction techniques are typically only capable of detecting and correcting non-word errors, such as typos and misspellings. However, context-sensitive errors, also known as real-word errors, are more challenging to detect because they are valid words that are used incorrectly in a given context. The Persian language, characterized by its rich morphology and complex syntax, presents formidable challenges to automatic spelling correction systems. Furthermore, the limited availability of Persian language resources makes it difficult to train effective spelling correction models. This paper introduces a cutting-edge approach for precise and efficient real-word error correction in Persian text. Our methodology adopts a structured, multi-tiered approach, employing semantic analysis, feature selection, and advanced classifiers to enhance error detection and correction efficacy. The innovative architecture discovers and stores semantic similarities between words and phrases in Persian text. The classifiers accurately identify real-word errors, while the semantic ranking algorithm determines the most probable corrections for real-word errors, taking into account specific spelling correction and context properties such as context, semantic similarity, and edit-distance measures. Evaluations have demonstrated that our proposed method surpasses previous Persian real-word error correction models. Our method achieves an impressive F-measure of 96.6% in the detection phase and an accuracy of 99.1% in the correction phase. These results clearly indicate that our approach is a highly promising solution for automatic real-word error correction in Persian text.
WiFaKey: Generating Cryptographic Keys from Face in the Wild
Venue: arXiv cs.CR 2024
Year: 2024

Deriving a unique cryptographic key from biometric measurements is a challenging task due to the existing noise gap between the biometric measurements and error correction coding. Additionally, privacy and security concerns arise as biometric measurements are inherently linked to the user. Biocryptosystems represent a key branch of solutions aimed at addressing these issues. However, many existing bio-cryptosystems rely on handcrafted feature extractors and error correction codes (ECC), often leading to performance degradation. To address these challenges and improve the reliability of biometric measurements, we propose a novel biometric cryptosystem named WiFaKey, for generating cryptographic keys from face in unconstrained settings. Speciffcally, WiFaKey ffrst introduces an adaptive random masking-driven feature transformation pipeline, AdaMTrans. AdaMTrans effectively quantizes and binarizes realvalued features and incorporates an adaptive random masking scheme to align the bit error rate with error correction requirements, thereby mitigating the noise gap. Besides, WiFaKey incorporates a supervised learning-based neural decoding scheme called Neural-MS decoder, which delivers a more robust error correction performance with less iteration than non-learning decoders, thereby alleviating the performance degradation. We evaluated WiFaKey using widely adopted face feature extractors on six large unconstrained and two constrained datasets. On the LFW dataset, WiFaKey achieved an average Genuine Match Rate of 85.45% and 85.20% at a 0% False Match Rate for MagFace and AdaFace features, respectively. Our comprehensive comparative analysis shows a signiffcant performance improvement of WiFaKey. The source code of our work is available at github.com/xingbod/WiFaKey.
Seal: Advancing Speech Language Models to be Few-Shot Learners
Venue: arXiv cs.CL 2024
Year: 2024

Existing auto-regressive language models have demonstrated a remarkable capability to perform a new task with just a few examples in prompt, without requiring any additional training. In order to extend this capability to a multi-modal setting (i.e. speech and language), this paper introduces the Seal model, an abbreviation for speech language model. It incorporates a novel alignment method, in which Kullback-Leibler divergence loss is performed to train a projector that bridges a frozen speech encoder with a frozen language model decoder. The resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks. Additionally, consistency experiments are conducted to validate its robustness on different pre-trained language models.
Latent Pollution Model: The Hidden Carbon Footprint in 3D Image Synthesis
Venue: arXiv cs.LG 2024
Year: 2024

Contemporary developments in generative AI are rapidly transforming the field of medical AI. These developments have been predominantly driven by the availability of large datasets and high computing power, which have facilitated a significant increase in model capacity. Despite their considerable potential, these models demand substantially high power, leading to high carbon dioxide (CO2) emissions. Given the harm such models are causing to the environment, there has been little focus on the carbon footprints of such models. This study analyzes carbon emissions from 2D and 3D latent diffusion models (LDMs) during training and data generation phases, revealing a surprising finding: the synthesis of large images contributes most significantly to these emissions. We assess different scenarios including model sizes, image dimensions, distributed training, and data generation steps. Our findings reveal substantial carbon emissions from these models, with training 2D and 3D models comparable to driving a car for 10 km and 90 km, respectively. The process of data generation is even more significant, with CO2 emissions equivalent to driving 160 km for 2D models and driving for up to 3345 km for 3D synthesis. Additionally, we found that the location of the experiment can increase carbon emissions by up to 94 times, and even the time of year can influence emissions by up to 50%. These figures are alarming, considering they represent only a single training and data generation phase for each model. Our results emphasize the urgent need for developing environmentally sustainable strategies in generative AI.
Automatic Generation of Fashion Images using Prompting in Generative Machine Learning Models
Venue: arXiv cs.CV 2024
Year: 2024

The advent of artificial intelligence has contributed in a groundbreaking transformation of the fashion industry, redefining creativity and innovation in unprecedented ways. This work investigates methodologies for generating tailored fashion descriptions using two distinct Large Language Models and a Stable Diffusion model for fashion image creation. Emphasizing adaptability in AI-driven fashion creativity, we depart from traditional approaches and focus on prompting techniques, such as zero-shot and few-shot learning, as well as Chain-of-Thought (CoT), which results in a variety of colors and textures, enhancing the diversity of the outputs. Central to our methodology is Retrieval-Augmented Generation (RAG), enriching models with insights from fashion sources to ensure contemporary representations. Evaluation combines quantitative metrics such as CLIPscore with qualitative human judgment, highlighting strengths in creativity, coherence, and aesthetic appeal across diverse styles. Among the participants, RAG and few-shot learning techniques are preferred for their ability to produce more relevant and appealing fashion descriptions. Our code is provided at https://github.com/georgiarg/AutoFashion.