OpenRaiser/Intern-Atlas huggingface dataset:
Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion
Venue: arXiv cs.CV 2024
Year: 2024

We present a method for generating Streetscapes-long sequences of views through an on-the-fly synthesized city-scale scene. Our generation is conditioned by language input (e.g., city name, weather), as well as an underlying map/layout hosting the desired trajectory. Compared to recent models for video generation or 3D view synthesis, our method can scale to much longer-range camera trajectories, spanning several city blocks, while maintaining visual quality and consistency. To achieve this goal, we build on recent work on video diffusion, used within an autoregressive framework that can easily scale to long sequences. In particular, we introduce a new temporal imputation method that prevents our autoregressive approach from drifting from the distribution of realistic city imagery. We train our Streetscapes system on a compelling source of data-posed imagery from Google Street View, along with contextual map data-which allows users to generate city views conditioned on any desired city layout, with controllable camera poses. Please see more results at our project page at https://boyangdeng.com/streetscapes.
Addressing Imbalance for Class Incremental Learning in Medical Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

Deep convolutional neural networks have made significant breakthroughs in medical image classification, under the assumption that training samples from all classes are simultaneously available. However, in real-world medical scenarios, there's a common need to continuously learn about new diseases, leading to the emerging field of class incremental learning (CIL) in the medical domain. Typically, CIL suffers from catastrophic forgetting when trained on new classes. This phenomenon is mainly caused by the imbalance between old and new classes, and it becomes even more challenging with imbalanced medical datasets. In this work, we introduce two simple yet effective plug-in methods to mitigate the adverse effects of the imbalance. First, we propose a CIL-balanced classification loss to mitigate the classifier bias toward majority classes via logit adjustment. Second, we propose a distribution margin loss that not only alleviates the inter-class overlap in embedding space but also enforces the intra-class compactness. We evaluate the effectiveness of our method with extensive experiments on three benchmark datasets (CCH5000, HAM10000, and EyePACS). The results demonstrate that our approach outperforms state-of-the-art methods.
Enhancing Software Supply Chain Resilience: Strategy For Mitigating Software Supply Chain Security Risks And Ensuring Security Continuity In Development Lifecycle
Venue: arXiv cs.CR 2024
Year: 2024

This article delves into the strategic approaches and preventive measures necessary to safeguard the software supply chain against evolving threats. It aims to foster an understanding of the challenges and vulnerabilities inherent in software supply chain resilience and to promote transparency and trust in the digital infrastructure that underpins contemporary society. By examining the concept of software supply chain resilience and assessing the current state of supply chain security, the article provides a foundation for discussing strategies and practices that can mitigate security risks and ensure security continuity throughout the development lifecycle. Through this comprehensive analysis, the article contributes to the ongoing effort to strengthen the security posture of software supply chains, thereby ensuring the reliable and secure operation of digital systems in a connected world
Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle
Venue: arXiv cs.CL 2024
Year: 2024

Recent innovations in language model training have demonstrated that it is possible to create highly performant models that are small enough to run on a smartphone. As these models are deployed in an increasing number of domains, it is critical to ensure that they are aligned with human preferences and safety considerations. In this report, we present our methodology for safety aligning the Phi-3 series of language models. We utilized a "break-fix" cycle, performing multiple rounds of dataset curation, safety post-training, benchmarking, red teaming, and vulnerability identification to cover a variety of harm areas in both single and multi-turn scenarios. Our results indicate that this approach iteratively improved the performance of the Phi-3 models across a wide range of responsible AI benchmarks. Finally, we include additional red teaming strategies and evaluations that were used to test the safety behavior of Phi-3.5-mini and Phi-3.5-MoE, which were optimized for multilingual capabilities.
Temperature Distribution Prediction in Laser Powder Bed Fusion using Transferable and Scalable Graph Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

This study presents novel predictive models using Graph Neural Networks (GNNs) for simulating thermal dynamics in Laser Powder Bed Fusion (L-PBF) processes. By developing and validating Single-Laser GNN (SL-GNN) and Multi-Laser GNN (ML-GNN) surrogates, the research introduces a scalable data-driven approach that learns fundamental physics from small-scale Finite Element Analysis (FEA) simulations and applies them to larger domains. Achieving a Mean Absolute Percentage Error (MAPE) of 3.77% with the baseline SL-GNN model, GNNs effectively learn from high-resolution simulations and generalize well across larger geometries. The proposed models capture the complexity of the heat transfer process in L-PBF while significantly reducing computational costs. For example, a thermomechanical simulation for a 2 mm x 2 mm domain typically requires about 4 hours, whereas the SL-GNN model can predict thermal distributions almost instantly. Calibrating models to larger domains enhances predictive performance, with significant drops in MAPE for 3 mm x 3 mm and 4 mm x 4 mm domains, highlighting the scalability and efficiency of this approach. Additionally, models show a decreasing trend in Root Mean Square Error (RMSE) when tuned to larger domains, suggesting potential for becoming geometry-agnostic. The interaction of multiple lasers complicates heat transfer, necessitating larger model architectures and advanced feature engineering. Using hyperparameters from Gaussian process-based Bayesian optimization, the best ML-GNN model demonstrates a 46.4% improvement in MAPE over the baseline ML-GNN model. In summary, this approach enables more efficient and flexible predictive modeling in L-PBF additive manufacturing.
SecureVAX: A Blockchain-Enabled Secure Vaccine Passport System
Venue: arXiv cs.CR 2024
Year: 2024

A vaccine passport serves as documentary proof, providing passport holders with greater freedom while roaming around during pandemics. It confirms vaccination against certain infectious diseases like COVID-19, Ebola, and flu. The key challenges faced by the digital vaccine passport system include passport forgery, unauthorized data access, and inaccurate information input by vaccination centers. Privacy concerns also need to be addressed to ensure that the user's personal identification information (PII) is not compromised. Additionally, it is necessary to track vaccine vials or doses to verify their authenticity, prevent misuse and illegal sales, as well as to restrict the illicit distribution of vaccines. To address these challenges, we propose a Blockchain-Enabled Secure Vaccine Passport System, leveraging the power of smart contracts. Our solution integrates off-chain and on-chain cryptographic computations, facilitating secure communication among various entities. We have utilized the InterPlanetary File System (IPFS) to store encrypted vaccine passports of citizens securely. Our prototype is built on the Ethereum platform, with smart contracts deployed on the Sepolia Test network, allowing for performance evaluation and validation of the system's effectiveness. By combining IPFS as a distributed data storage platform and Ethereum as a blockchain platform, our solution paves the way for secure, efficient, and globally interoperable vaccine passport management, supporting comprehensive vaccination initiatives worldwide.
Data-Algorithm-Architecture Co-Optimization for Fair Neural Networks on Skin Lesion Dataset
Venue: arXiv cs.LG 2024
Year: 2024

As Artificial Intelligence (AI) increasingly integrates into our daily lives, fairness has emerged as a critical concern, particularly in medical AI, where datasets often reflect inherent biases due to social factors like the underrepresentation of marginalized communities and socioeconomic barriers to data collection. Traditional approaches to mitigating these biases have focused on data augmentation and the development of fairness-aware training algorithms. However, this paper argues that the architecture of neural networks, a core component of Machine Learning (ML), plays a crucial role in ensuring fairness. We demonstrate that addressing fairness effectively requires a holistic approach that simultaneously considers data, algorithms, and architecture. Utilizing Automated ML (AutoML) technology, specifically Neural Architecture Search (NAS), we introduce a novel framework, BiaslessNAS, designed to achieve fair outcomes in analyzing skin lesion datasets. BiaslessNAS incorporates fairness considerations at every stage of the NAS process, leading to the identification of neural networks that are not only more accurate but also significantly fairer. Our experiments show that BiaslessNAS achieves a 2.55% increase in accuracy and a 65.50% improvement in fairness compared to traditional NAS methods, underscoring the importance of integrating fairness into neural network architecture for better outcomes in medical AI applications.
Synthetic Counterfactual Faces
Venue: arXiv cs.CV 2024
Year: 2024

Computer vision systems have been deployed in various applications involving biometrics like human faces. These systems can identify social media users, search for missing persons, and verify identity of individuals. While computer vision models are often evaluated for accuracy on available benchmarks, more annotated data is necessary to learn about their robustness and fairness against semantic distributional shifts in input data, especially in face data. Among annotated data, counterfactual examples grant strong explainability characteristics. Because collecting natural face data is prohibitively expensive, we put forth a generative AI-based framework to construct targeted, counterfactual, high-quality synthetic face data. Our synthetic data pipeline has many use cases, including face recognition systems sensitivity evaluations and image understanding system probes. The pipeline is validated with multiple user studies. We showcase the efficacy of our face generation pipeline on a leading commercial vision model. We identify facial attributes that cause vision systems to fail.
Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction
Venue: arXiv cs.CL 2024
Year: 2024

This paper introduces Werewolf Arena, a novel framework for evaluating large language models (LLMs) through the lens of the classic social deduction game, Werewolf. In Werewolf Arena, LLMs compete against each other, navigating the game's complex dynamics of deception, deduction, and persuasion. The framework introduces a dynamic turn-taking system based on bidding, mirroring real-world discussions where individuals strategically choose when to speak. We demonstrate the framework's utility through an arena-style tournament featuring Gemini and GPT models. Our results reveal distinct strengths and weaknesses in the models' strategic reasoning and communication. These findings highlight Werewolf Arena's potential as a challenging and scalable LLM benchmark.
Persuading while Learning
Venue: arXiv cs.GT 2024
Year: 2024

We propose a dynamic product adoption persuasion model involving an impatient partially informed sender who gradually learns the state. In this model, the sender gathers information over time, and hence her posteriors' sequence forms a discrete-time martingale. The sender commits to a dynamic revelation policy to persuade the agent to adopt a product. We demonstrate that under the assumption that the sender's martingale possesses Blackwell-preserving kernels, the family of optimal strategies for the sender takes an interval form; namely, in every period the set of martingale realizations in which adoption occurs is an interval. Utilizing this, we prove that if the sender is sufficiently impatient, then under a random walk martingale, the optimal policy is fully transparent up to the moment of adoption; namely, the sender reveals the entire information she privately holds in every period.
LLAssist: Simple Tools for Automating Literature Review Using Large Language Models
Venue: arXiv cs.DL 2024
Year: 2024

This paper introduces LLAssist, an open-source tool designed to streamline literature reviews in academic research. In an era of exponential growth in scientific publications, researchers face mounting challenges in efficiently processing vast volumes of literature. LLAssist addresses this issue by leveraging Large Language Models (LLMs) and Natural Language Processing (NLP) techniques to automate key aspects of the review process. Specifically, it extracts important information from research articles and evaluates their relevance to user-defined research questions. The goal of LLAssist is to significantly reduce the time and effort required for comprehensive literature reviews, allowing researchers to focus more on analyzing and synthesizing information rather than on initial screening tasks. By automating parts of the literature review workflow, LLAssist aims to help researchers manage the growing volume of academic publications more efficiently.
On the normality of Boolean quartics
Venue: arXiv cs.DM 2024
Year: 2024

In the BFA 2023 conference paper, A. Polujan, L. Mariot and S. Picek exhibited the first example of a non-normal but weakly normal bent function in dimension 8. In this note, we present numerical approaches based on the classification of Boolean spaces to explore in detail the normality of bent functions of 8 variables and we complete S. Dubuc s results for dimensions less or equal to 7. Based on our investigations, we show that all bent functions in 8 variables are normal or weakly normal. Finally, we conjecture that more generally all Boolean functions of degree at most 4 in 8 variables are normal or weakly normal.
BERTer: The Efficient One
Venue: arXiv cs.CL 2024
Year: 2024

We explore advanced fine-tuning techniques to boost BERT's performance in sentiment analysis, paraphrase detection, and semantic textual similarity. Our approach leverages SMART regularization to combat overfitting, improves hyperparameter choices, employs a cross-embedding Siamese architecture for improved sentence embeddings, and introduces innovative early exiting methods. Our fine-tuning findings currently reveal substantial improvements in model efficiency and effectiveness when combining multiple fine-tuning architectures, achieving a state-of-the-art performance score of on the test set, surpassing current benchmarks and highlighting BERT's adaptability in multifaceted linguistic tasks.
FAVis: Visual Analytics of Factor Analysis for Psychological Research
Venue: arXiv cs.HC 2024
Year: 2024

Psychological research often involves understanding psychological constructs through conducting factor analysis on data collected by a questionnaire, which can comprise hundreds of questions. Without interactive systems for interpreting factor models, researchers are frequently exposed to subjectivity, potentially leading to misinterpretations or overlooked crucial information. This paper introduces FAVis, a novel interactive visualization tool designed to aid researchers in interpreting and evaluating factor analysis results. FAVis enhances the understanding of relationships between variables and factors by supporting multiple views for visualizing factor loadings and correlations, allowing users to analyze information from various perspectives. The primary feature of FAVis is to enable users to set optimal thresholds for factor loadings to balance clarity and information retention. FAVis also allows users to assign tags to variables, enhancing the understanding of factors by linking them to their associated psychological constructs. Our user study demonstrates the utility of FAVis in various tasks.
LoAS: Fully Temporal-Parallel Dataflow for Dual-Sparse Spiking Neural Networks
Venue: arXiv cs.AR 2024
Year: 2024

Spiking Neural Networks (SNNs) have gained significant research attention in the last decade due to their potential to drive resource-constrained edge devices. Though existing SNN accelerators offer high efficiency in processing sparse spikes with dense weights, opportunities are less explored in SNNs with sparse weights, i.e., dual-sparsity. In this work, we study the acceleration of dual-sparse SNNs, focusing on their core operation, sparse-matrix-sparse-matrix multiplication (spMspM). We observe that naively running a dual-sparse SNN on existing spMspM accelerators designed for dual-sparse Artificial Neural Networks (ANNs) exhibits sub-optimal efficiency. The main challenge is that processing timesteps, a natural property of SNNs, introduces an extra loop to ANN spMspM, leading to longer latency and more memory traffic. To address the problem, we propose a fully temporal-parallel (FTP) dataflow, which minimizes both data movement across timesteps and the end-to-end latency of dual-sparse SNNs. To maximize the efficiency of FTP dataflow, we propose an FTP-friendly spike compression mechanism that efficiently compresses single-bit spikes and ensures contiguous memory access. We further propose an FTP-friendly inner-join circuit that can lower the cost of the expensive prefix-sum circuits with almost no throughput penalty. All the above techniques for FTP dataflow are encapsulated in LoAS, a Low-latency inference Accelerator for dual-sparse SNNs. With FTP dataflow, compression, and inner-join, running dual-sparse SNN workloads on LoAS demonstrates significant speedup (up to $8.51\times$) and energy reduction (up to $3.68\times$) compared to running it on prior dual-sparse accelerators.
Stable-Hair: Real-World Hair Transfer via Diffusion Model
Venue: arXiv cs.CV 2024
Year: 2024

Current hair transfer methods struggle to handle diverse and intricate hairstyles, limiting their applicability in real-world scenarios. In this paper, we propose a novel diffusion-based hair transfer framework, named \textit{Stable-Hair}, which robustly transfers a wide range of real-world hairstyles to user-provided faces for virtual hair try-on. To achieve this goal, our Stable-Hair framework is designed as a two-stage pipeline. In the first stage, we train a Bald Converter alongside stable diffusion to remove hair from the user-provided face images, resulting in bald images. In the second stage, we specifically designed a Hair Extractor and a Latent IdentityNet to transfer the target hairstyle with highly detailed and high-fidelity to the bald image. The Hair Extractor is trained to encode reference images with the desired hairstyles, while the Latent IdentityNet ensures consistency in identity and background. To minimize color deviations between source images and transfer results, we introduce a novel Latent ControlNet architecture, which functions as both the Bald Converter and Latent IdentityNet. After training on our curated triplet dataset, our method accurately transfers highly detailed and high-fidelity hairstyles to the source images. Extensive experiments demonstrate that our approach achieves state-of-the-art performance compared to existing hair transfer methods. Project page: \textcolor{red}{\url{https://xiaojiu-z.github.io/Stable-Hair.github.io/}}
Stochastic Distance in Property Testing
Venue: arXiv cs.DC 2024
Year: 2024

We introduce a novel concept termed "stochastic distance" for property testing. Diverging from the traditional definition of distance, where a distance $t$ implies that there exist $t$ edges that can be added to ensure a graph possesses a certain property (such as $k$-edge-connectivity), our new notion implies that there is a high probability that adding $t$ random edges will endow the graph with the desired property. While formulating testers based on this new distance proves challenging in a sequential environment, it is much easier in a distributed setting. Taking $k$-edge-connectivity as a case study, we design ultra-fast testing algorithms in the CONGEST model. Our introduction of stochastic distance offers a more natural fit for the distributed setting, providing a promising avenue for future research in emerging models of computation.
A3Rank: Augmentation Alignment Analysis for Prioritizing Overconfident Failing Samples for Deep Learning Models
Venue: arXiv cs.SE 2024
Year: 2024

Sharpening deep learning models by training them with examples close to the decision boundary is a well-known best practice. Nonetheless, these models are still error-prone in producing predictions. In practice, the inference of the deep learning models in many application systems is guarded by a rejector, such as a confidence-based rejector, to filter out samples with insufficient prediction confidence. Such confidence-based rejectors cannot effectively guard against failing samples with high confidence. Existing test case prioritization techniques effectively distinguish confusing samples from confident samples to identify failing samples among the confusing ones, yet prioritizing the failing ones high among many confident ones is challenging. In this paper, we propose $A^3$Rank, a novel test case prioritization technique with augmentation alignment analysis, to address this problem. $A^3$Rank generates augmented versions of each test case and assesses the extent of the prediction result for the test case misaligned with these of the augmented versions and vice versa. Our experiment shows that $A^3$Rank can effectively rank failing samples escaping from the checking of confidence-based rejectors, which significantly outperforms the peer techniques by 163.63\% in the detection ratio of top-ranked samples. We also provide a framework to construct a detector devoted to augmenting these rejectors to defend these failing samples, and our detector can achieve a significantly higher defense success rate.
Shape and Style GAN-based Multispectral Data Augmentation for Crop/Weed Segmentation in Precision Farming
Venue: arXiv cs.CV 2024
Year: 2024

The use of deep learning methods for precision farming is gaining increasing interest. However, collecting training data in this application field is particularly challenging and costly due to the need of acquiring information during the different growing stages of the cultivation of interest. In this paper, we present a method for data augmentation that uses two GANs to create artificial images to augment the training data. To obtain a higher image quality, instead of re-creating the entire scene, we take original images and replace only the patches containing objects of interest with artificial ones containing new objects with different shapes and styles. In doing this, we take into account both the foreground (i.e., crop samples) and the background (i.e., the soil) of the patches. Quantitative experiments, conducted on publicly available datasets, demonstrate the effectiveness of the proposed approach. The source code and data discussed in this work are available as open source.