OpenRaiser/Intern-Atlas huggingface dataset:
Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback
Venue: arXiv cs.LG 2024
Year: 2024

Latent diffusion models are the state-of-the-art for synthetic image generation. To align these models with human preferences, training the models using reinforcement learning on human feedback is crucial. Black et. al 2024 introduced denoising diffusion policy optimisation (DDPO), which accounts for the iterative denoising nature of the generation by modelling it as a Markov chain with a final reward. As the reward is a single value that determines the model's performance on the entire image, the model has to navigate a very sparse reward landscape and so requires a large sample count. In this work, we extend the DDPO by presenting the Pixel-wise Policy Optimisation (PXPO) algorithm, which can take feedback for each pixel, providing a more nuanced reward to the model.
ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editing
Venue: arXiv cs.CV 2024
Year: 2024

Recently, researchers have proposed powerful systems for generating and manipulating images using natural language instructions. However, it is difficult to precisely specify many common classes of image transformations with text alone. For example, a user may wish to change the location and breed of a particular dog in an image with several similar dogs. This task is quite difficult with natural language alone, and would require a user to write a laboriously complex prompt that both disambiguates the target dog and describes the destination. We propose ClickDiffusion, a system for precise image manipulation and generation that combines natural language instructions with visual feedback provided by the user through a direct manipulation interface. We demonstrate that by serializing both an image and a multi-modal instruction into a textual representation it is possible to leverage LLMs to perform precise transformations of the layout and appearance of an image. Code available at https://github.com/poloclub/ClickDiffusion.
LOSS-SLAM: Lightweight Open-Set Semantic Simultaneous Localization and Mapping
Venue: arXiv cs.RO 2024
Year: 2024

Enabling robots to understand the world in terms of objects is a critical building block towards higher level autonomy. The success of foundation models in vision has created the ability to segment and identify nearly all objects in the world. However, utilizing such objects to localize the robot and build an open-set semantic map of the world remains an open research question. In this work, a system of identifying, localizing, and encoding objects is tightly coupled with probabilistic graphical models for performing open-set semantic simultaneous localization and mapping (SLAM). Results are presented demonstrating that the proposed lightweight object encoding can be used to perform more accurate object-based SLAM than existing open-set methods, closed-set methods, and geometric methods while incurring a lower computational overhead than existing open-set mapping methods.
"It is okay to be uncommon": Quantizing Sound Event Detection Networks on Hardware Accelerators with Uncommon Sub-Byte Support
Venue: arXiv cs.SD 2024
Year: 2024

If our noise-canceling headphones can understand our audio environments, they can then inform us of important sound events, tune equalization based on the types of content we listen to, and dynamically adjust noise cancellation parameters based on audio scenes to further reduce distraction. However, running multiple audio understanding models on headphones with a limited energy budget and on-chip memory remains a challenging task. In this work, we identify a new class of neural network accelerators (e.g., NE16 on GAP9) that allows network weights to be quantized to different common (e.g., 8 bits) and uncommon bit-widths (e.g., 3 bits). We then applied a differentiable neural architecture search to search over the optimal bit-widths of a network on two different sound event detection tasks with potentially different requirements on quantization and prediction granularity (i.e., classification vs. embeddings for few-shot learning). We further evaluated our quantized models on actual hardware, showing that we reduce memory usage, inference latency, and energy consumption by an average of 62%, 46%, and 61% respectively compared to 8-bit models while maintaining floating point performance. Our work sheds light on the benefits of such accelerators on sound event detection tasks when combined with an appropriate search method.
Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomes
Venue: arXiv cs.CR 2024
Year: 2024

Large Language Models (LLMs) have gained widespread adoption across various domains, including chatbots and auto-task completion agents. However, these models are susceptible to safety vulnerabilities such as jailbreaking, prompt injection, and privacy leakage attacks. These vulnerabilities can lead to the generation of malicious content, unauthorized actions, or the disclosure of confidential information. While foundational LLMs undergo alignment training and incorporate safety measures, they are often subject to fine-tuning, or doing quantization resource-constrained environments. This study investigates the impact of these modifications on LLM safety, a critical consideration for building reliable and secure AI systems. We evaluate foundational models including Mistral, Llama series, Qwen, and MosaicML, along with their fine-tuned variants. Our comprehensive analysis reveals that fine-tuning generally increases the success rates of jailbreak attacks, while quantization has variable effects on attack success rates. Importantly, we find that properly implemented guardrails significantly enhance resistance to jailbreak attempts. These findings contribute to our understanding of LLM vulnerabilities and provide insights for developing more robust safety strategies in the deployment of language models.
A Ground Mobile Robot for Autonomous Terrestrial Laser Scanning-Based Field Phenotyping
Venue: arXiv cs.RO 2024
Year: 2024

Traditional field phenotyping methods are often manual, time-consuming, and destructive, posing a challenge for breeding progress. To address this bottleneck, robotics and automation technologies offer efficient sensing tools to monitor field evolution and crop development throughout the season. This study aimed to develop an autonomous ground robotic system for LiDAR-based field phenotyping in plant breeding trials. A Husky platform was equipped with a high-resolution three-dimensional (3D) laser scanner to collect in-field terrestrial laser scanning (TLS) data without human intervention. To automate the TLS process, a 3D ray casting analysis was implemented for optimal TLS site planning, and a route optimization algorithm was utilized to minimize travel distance during data collection. The platform was deployed in two cotton breeding fields for evaluation, where it autonomously collected TLS data. The system provided accurate pose information through RTK-GNSS positioning and sensor fusion techniques, with average errors of less than 0.6 cm for location and 0.38$^{\circ}$ for heading. The achieved localization accuracy allowed point cloud registration with mean point errors of approximately 2 cm, comparable to traditional TLS methods that rely on artificial targets and manual sensor deployment. This work presents an autonomous phenotyping platform that facilitates the quantitative assessment of plant traits under field conditions of both large agricultural fields and small breeding trials to contribute to the advancement of plant phenomics and breeding programs.
AI Knowledge and Reasoning: Emulating Expert Creativity in Scientific Research
Venue: arXiv cs.AI 2024
Year: 2024

We investigate whether modern AI can emulate expert creativity in complex scientific endeavors. We introduce novel methodology that utilizes original research articles published after the AI's training cutoff, ensuring no prior exposure, mitigating concerns of rote memorization and prior training. The AI are tasked with redacting findings, predicting outcomes from redacted research, and assessing prediction accuracy against reported results. Analysis on 589 published studies in four leading psychology journals over a 28-month period, showcase the AI's proficiency in understanding specialized research, deductive reasoning, and evaluating evidentiary alignment--cognitive hallmarks of human subject matter expertise and creativity. These findings suggest the potential of general-purpose AI to transform academia, with roles requiring knowledge-based creativity become increasingly susceptible to technological substitution.
Beyond the Known: Adversarial Autoencoders in Novelty Detection
Venue: arXiv cs.CV 2024
Year: 2024

In novelty detection, the goal is to decide if a new data point should be categorized as an inlier or an outlier, given a training dataset that primarily captures the inlier distribution. Recent approaches typically use deep encoder and decoder network frameworks to derive a reconstruction error, and employ this error either to determine a novelty score, or as the basis for a one-class classifier. In this research, we use a similar framework but with a lightweight deep network, and we adopt a probabilistic score with reconstruction error. Our methodology calculates the probability of whether the sample comes from the inlier distribution or not. This work makes two key contributions. The first is that we compute the novelty probability by linearizing the manifold that holds the structure of the inlier distribution. This allows us to interpret how the probability is distributed and can be determined in relation to the local coordinates of the manifold tangent space. The second contribution is that we improve the training protocol for the network. Our results indicate that our approach is effective at learning the target class, and it outperforms recent state-of-the-art methods on several benchmark datasets.
Comparison of algorithms in Foreign Exchange Rate Prediction
Venue: arXiv cs.CV 2024
Year: 2024

Foreign currency exchange plays a vital role for trading of currency in the financial market. Due to its volatile nature, prediction of foreign currency exchange is a challenging task. This paper presents different machine learning techniques like Artificial Neural Network (ANN), Recurrent Neural Network (RNN) to develop prediction model between Nepalese Rupees against three major currencies Euro, Pound Sterling and US dollar. Recurrent Neural Network is a type of neural network that have feedback connections. In this paper, prediction model were based on different RNN architectures, feed forward ANN with back propagation algorithm and then compared the accuracy of each model. Different ANN architecture models like Feed forward neural network, Simple Recurrent Neural Network (SRNN), Gated Recurrent Unit (GRU) and Long Short Term Memory (LSTM) were used. Input parameters were open, low, high and closing prices for each currency. From this study, we have found that LSTM networks provided better results than SRNN and GRU networks.
Fundamental Limits of Optical Fiber MIMO Channels With Finite Blocklength
Venue: arXiv cs.IT 2024
Year: 2024

The multiple-input and multiple-output (MIMO) technique is regarded as a promising approach to boost the throughput and reliability of optical fiber communications. However, the fundamental limits of optical fiber MIMO systems with finite block-length (FBL) are not available in the literature. This paper studies the fundamental limits of optical fiber multicore/multimode systems in the FBL regime when the coding rate is a perturbation within $\mathcal{O}(\frac{1}{\sqrt{ML}})$ of the capacity, where M and L represent the number of transmit channels and blocklength, respectively. Considering the Jacobi MIMO channel, which was proposed to model the nearly lossless propagation and the crosstalks in optical fiber systems, we derive the upper and lower bounds for the optimal error probability. For that purpose, we first set up the central limit theorem for the information density in the asymptotic regime where the number of transmit, receive, available channels and the blocklength go to infinity at the same pace. The result is then utilized to derive the upper and lower bounds for the optimal average error probability with the concerned rate. The derived theoretical results reveal interesting physical insights for Jacobi MIMO channels with FBL. First, the derived bounds for Jacobi channels degenerate to those for Rayleigh channels when the number of available channels approaches infinity. Second, the high signal-to-noise (SNR) approximation indicates that a larger number of available channels results in a larger error probability. Numerical results validate the accuracy of the theoretical results and show that the derived bounds are closer to the performance of practical LDPC codes than outage probability.
Joint Identifiability of Cross-Domain Recommendation via Hierarchical Subspace Disentanglement
Venue: arXiv cs.IR 2024
Year: 2024

Cross-Domain Recommendation (CDR) seeks to enable effective knowledge transfer across domains. Existing works rely on either representation alignment or transformation bridges, but they struggle on identifying domain-shared from domain-specific latent factors. Specifically, while CDR describes user representations as a joint distribution over two domains, these methods fail to account for its joint identifiability as they primarily fixate on the marginal distribution within a particular domain. Such a failure may overlook the conditionality between two domains and how it contributes to latent factor disentanglement, leading to negative transfer when domains are weakly correlated. In this study, we explore what should and should not be transferred in cross-domain user representations from a causality perspective. We propose a Hierarchical subspace disentanglement approach to explore the Joint IDentifiability of cross-domain joint distribution, termed HJID, to preserve domain-specific behaviors from domain-shared factors. HJID organizes user representations into layers: generic shallow subspaces and domain-oriented deep subspaces. We first encode the generic pattern in the shallow subspace by minimizing the Maximum Mean Discrepancy of initial layer activation. Then, to dissect how domain-oriented latent factors are encoded in deeper layers activation, we construct a cross-domain causality-based data generation graph, which identifies cross-domain consistent and domain-specific components, adhering to the Minimal Change principle. This allows HJID to maintain stability whilst discovering unique factors for different domains, all within a generative framework of invertible transformations that guarantee the joint identifiability. With experiments on real-world datasets, we show that HJID outperforms SOTA methods on a range of strongly and weakly correlated CDR tasks.
Exploring UAV Networking from the Terrain Information Completeness Perspective: A Tutorial
Venue: arXiv cs.NI 2024
Year: 2024

Terrain information is a crucial factor affecting the performance of unmanned aerial vehicle (UAV) networks. As a tutorial, this article provides a unique perspective on the completeness of terrain information, summarizing and enhancing the research on terrain-based UAV deployment. In the presence of complete terrain information, two highly discussed topics are UAV-aided map construction and dynamic trajectory design based on maps. We propose a case study illustrating the mutually reinforcing relationship between them. When terrain information is incomplete, and only terrain-related feature parameters are available, we discuss how existing models map terrain features to blockage probabilities. By introducing the application of this model with stochastic geometry, a case study is proposed to analyze the accuracy of the model. When no terrain information is available, UAVs gather terrain information during the real-time networking process and determine the next position by collected information. This real-time search method is currently limited to relay communication. In the case study, we extend it to a multi-user scenario and summarize three trade-offs of the method. Finally, we conduct a qualitative analysis to assess the impact of three factors that have been overlooked in terrain-based UAV deployment.
Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Unsupervised domain adaptation (UDA) techniques are vital for semantic segmentation in geosciences, effectively utilizing remote sensing imagery across diverse domains. However, most existing UDA methods, which focus on domain alignment at the high-level feature space, struggle to simultaneously retain local spatial details and global contextual semantics. To overcome these challenges, a novel decomposition scheme is proposed to guide domain-invariant representation learning. Specifically, multiscale high/low-frequency decomposition (HLFD) modules are proposed to decompose feature maps into high- and low-frequency components across different subspaces. This decomposition is integrated into a fully global-local generative adversarial network (GLGAN) that incorporates global-local transformer blocks (GLTBs) to enhance the alignment of decomposed features. By integrating the HLFD scheme and the GLGAN, a novel decomposition-based UDA framework called De-GLGAN is developed to improve the cross-domain transferability and generalization capability of semantic segmentation models. Extensive experiments on two UDA benchmarks, namely ISPRS Potsdam and Vaihingen, and LoveDA Rural and Urban, demonstrate the effectiveness and superiority of the proposed approach over existing state-of-the-art UDA methods. The source code for this work is accessible at https://github.com/sstary/SSRS.
Exhaustive Exploitation of Nature-inspired Computation for Cancer Screening in an Ensemble Manner
Venue: arXiv cs.NE 2024
Year: 2024

Accurate screening of cancer types is crucial for effective cancer detection and precise treatment selection. However, the association between gene expression profiles and tumors is often limited to a small number of biomarker genes. While computational methods using nature-inspired algorithms have shown promise in selecting predictive genes, existing techniques are limited by inefficient search and poor generalization across diverse datasets. This study presents a framework termed Evolutionary Optimized Diverse Ensemble Learning (EODE) to improve ensemble learning for cancer classification from gene expression data. The EODE methodology combines an intelligent grey wolf optimization algorithm for selective feature space reduction, guided random injection modeling for ensemble diversity enhancement, and subset model optimization for synergistic classifier combinations. Extensive experiments were conducted across 35 gene expression benchmark datasets encompassing varied cancer types. Results demonstrated that EODE obtained significantly improved screening accuracy over individual and conventionally aggregated models. The integrated optimization of advanced feature selection, directed specialized modeling, and cooperative classifier ensembles helps address key challenges in current nature-inspired approaches. This provides an effective framework for robust and generalized ensemble learning with gene expression biomarkers. Specifically, we have opened EODE source code on Github at https://github.com/wangxb96/EODE.
NPB-REC: A Non-parametric Bayesian Deep-learning Approach for Undersampled MRI Reconstruction with Uncertainty Estimation
Venue: arXiv cs.CV 2024
Year: 2024

The ability to reconstruct high-quality images from undersampled MRI data is vital in improving MRI temporal resolution and reducing acquisition times. Deep learning methods have been proposed for this task, but the lack of verified methods to quantify the uncertainty in the reconstructed images hampered clinical applicability. We introduce "NPB-REC", a non-parametric fully Bayesian framework, for MRI reconstruction from undersampled data with uncertainty estimation. We use Stochastic Gradient Langevin Dynamics during training to characterize the posterior distribution of the network parameters. This enables us to both improve the quality of the reconstructed images and quantify the uncertainty in the reconstructed images. We demonstrate the efficacy of our approach on a multi-coil MRI dataset from the fastMRI challenge and compare it to the baseline End-to-End Variational Network (E2E-VarNet). Our approach outperforms the baseline in terms of reconstruction accuracy by means of PSNR and SSIM ($34.55$, $0.908$ vs. $33.08$, $0.897$, $p<0.01$, acceleration rate $R=8$) and provides uncertainty measures that correlate better with the reconstruction error (Pearson correlation, $R=0.94$ vs. $R=0.91$). Additionally, our approach exhibits better generalization capabilities against anatomical distribution shifts (PSNR and SSIM of $32.38$, $0.849$ vs. $31.63$, $0.836$, $p<0.01$, training on brain data, inference on knee data, acceleration rate $R=8$). NPB-REC has the potential to facilitate the safe utilization of deep learning-based methods for MRI reconstruction from undersampled data. Code and trained models are available at \url{https://github.com/samahkh/NPB-REC}.
EVT-enriched Radio Maps for URLLC
Venue: arXiv cs.NI 2024
Year: 2024

This paper introduces a sophisticated and adaptable framework combining extreme value theory with radio maps to spatially model extreme channel conditions accurately. Utilising existing signal-to-noise ratio (SNR) measurements and leveraging Gaussian processes, our approach predicts the tail of the SNR distribution, which entails estimating the parameters of a generalised Pareto distribution, at unobserved locations. This innovative method offers a versatile solution adaptable to various resource allocation challenges in ultra-reliable low-latency communications. We evaluate the performance of this method in a rate maximisation problem with defined outage constraints and compare it with a benchmark in the literature. Notably, the proposed approach meets the outage demands in a larger percentage of the coverage area and reaches higher transmission rates.
A Map of Exploring Human Interaction patterns with LLM: Insights into Collaboration and Creativity
Venue: arXiv cs.HC 2024
Year: 2024

The outstanding performance capabilities of large language model have driven the evolution of current AI system interaction patterns. This has led to considerable discussion within the Human-AI Interaction (HAII) community. Numerous studies explore this interaction from technical, design, and empirical perspectives. However, the majority of current literature reviews concentrate on interactions across the wider spectrum of AI, with limited attention given to the specific realm of interaction with LLM. We searched for articles on human interaction with LLM, selecting 110 relevant publications meeting consensus definition of Human-AI interaction. Subsequently, we developed a comprehensive Mapping Procedure, structured in five distinct stages, to systematically analyze and categorize the collected publications. Applying this methodical approach, we meticulously mapped the chosen studies, culminating in a detailed and insightful representation of the research landscape. Overall, our review presents an novel approach, introducing a distinctive mapping method, specifically tailored to evaluate human-LLM interaction patterns. We conducted a comprehensive analysis of the current research in related fields, employing clustering techniques for categorization, which enabled us to clearly delineate the status and challenges prevalent in each identified area.