OpenRaiser/Intern-Atlas huggingface dataset:
RadGazeGen: Radiomics and Gaze-guided Medical Image Generation using Diffusion Models
Venue: arXiv cs.CV 2024
Year: 2024

In this work, we present RadGazeGen, a novel framework for integrating experts' eye gaze patterns and radiomic feature maps as controls to text-to-image diffusion models for high fidelity medical image generation. Despite the recent success of text-to-image diffusion models, text descriptions are often found to be inadequate and fail to convey detailed disease-specific information to these models to generate clinically accurate images. The anatomy, disease texture patterns, and location of the disease are extremely important to generate realistic images; moreover the fidelity of image generation can have significant implications in downstream tasks involving disease diagnosis or treatment repose assessment. Hence, there is a growing need to carefully define the controls used in diffusion models for medical image generation. Eye gaze patterns of radiologists are important visuo-cognitive information, indicative of subtle disease patterns and spatial location. Radiomic features further provide important subvisual cues regarding disease phenotype. In this work, we propose to use these gaze patterns in combination with standard radiomics descriptors, as controls, to generate anatomically correct and disease-aware medical images. RadGazeGen is evaluated for image generation quality and diversity on the REFLACX dataset. To demonstrate clinical applicability, we also show classification performance on the generated images from the CheXpert test set (n=500) and long-tailed learning performance on the MIMIC-CXR-LT test set (n=23550).
Self-controller: Controlling LLMs with Multi-round Step-by-step Self-awareness
Venue: arXiv cs.CL 2024
Year: 2024

The applications of large language models (LLMs) have been widely spread across all domains. However, the basic abilities such as the controllability of LLMs are still limited. To address this, we propose "Self-controller", a novel agentic framework bringing self-awareness into LLMs' reasoning logic. The core idea of this work is to maintain states based on the LLM's response, letting the LLM become self-aware of current status and think step by step in a multi-round chain-of-thought paradigm. Our experiment on the state of textual length has shown the controllability and effectiveness of the Self-controller. We further implement a binary search algorithm to accelerate the generation process based on the linearity and monotonicity of the textual length state. Another advantage of the Self-controller comes with DeepSeek's Context Caching technology, which significantly saves computational token consumption when a cluster of conversations shares the same prefix of context. Theoretically, we prove that in this scenario the extra time complexity is $O(c \log n)$. Results of the back-of-the-envelope estimation suggest that the token consumption of our method is no more than twice as much as that of the trivial single-round generation. Furthermore, our ablation study on word constraints demonstrates the Self-controller's consistent controllability across all foundation models.
TFCT-I2P: Three stream fusion network with color aware transformer for image-to-point cloud registration
Venue: arXiv cs.CV 2024
Year: 2024

Along with the advancements in artificial intelligence technologies, image-to-point-cloud registration (I2P) techniques have made significant strides. Nevertheless, the dimensional differences in the features of points cloud (three-dimension) and image (two-dimension) continue to pose considerable challenges to their development. The primary challenge resides in the inability to leverage the features of one modality to augment those of another, thereby complicating the alignment of features within the latent space. To address this challenge, we propose an image-to-point-cloud method named as TFCT-I2P. Initially, we introduce a Three-Stream Fusion Network (TFN), which integrates color information from images with structural information from point clouds, facilitating the alignment of features from both modalities. Subsequently, to effectively mitigate patch-level misalignments introduced by the inclusion of color information, we design a Color-Aware Transformer (CAT). Finally, we conduct extensive experiments on 7Scenes, RGB-D Scenes V2, ScanNet V2, and a self-collected dataset. The results demonstrate that TFCT-I2P surpasses state-of-the-art methods by 1.5% in Inlier Ratio, 0.4% in Feature Matching Recall, and 5.4% in Registration Recall. Therefore, we believe that the proposed TFCT-I2P contributes to the advancement of I2P registration.
Guided Statistical Workflows with Interactive Explanations and Assumption Checking
Venue: arXiv cs.HC 2024
Year: 2024

Statistical practices such as building regression models or running hypothesis tests rely on following rigorous procedures of steps and verifying assumptions on data to produce valid results. However, common statistical tools do not verify users' decision choices and provide low-level statistical functions without instructions on the whole analysis practice. Users can easily misuse analysis methods, potentially decreasing the validity of results. To address this problem, we introduce GuidedStats, an interactive interface within computational notebooks that encapsulates guidance, models, visualization, and exportable results into interactive workflows. It breaks down typical analysis processes, such as linear regression and two-sample T-tests, into interactive steps supplemented with automatic visualizations and explanations for step-wise evaluation. Users can iterate on input choices to refine their models, while recommended actions and exports allow the user to continue their analysis in code. Case studies show how GuidedStats offers valuable instructions for conducting fluid statistical analyses while finding possible assumption violations in the underlying data, supporting flexible and accurate statistical analyses.
GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction
Venue: arXiv cs.CV 2024
Year: 2024

Image restoration and spectral reconstruction are longstanding computer vision tasks. Currently, CNN-transformer hybrid models provide state-of-the-art performance for these tasks. The key common ingredient in the architectural designs of these models is Channel-wise Self-Attention (CSA). We first show that CSA is an overall low-rank operation. Then, we propose an instance-Guided Low-rank Multi-Head selfattention (GLMHA) to replace the CSA for a considerable computational gain while closely retaining the original model performance. Unique to the proposed GLMHA is its ability to provide computational gain for both short and long input sequences. In particular, the gain is in terms of both Floating Point Operations (FLOPs) and parameter count reduction. This is in contrast to the existing popular computational complexity reduction techniques, e.g., Linformer, Performer, and Reformer, for whom FLOPs overpower the efficient design tricks for the shorter input sequences. Moreover, parameter reduction remains unaccounted for in the existing methods.We perform an extensive evaluation for the tasks of spectral reconstruction from RGB images, spectral reconstruction from snapshot compressive imaging, motion deblurring, and image deraining by enhancing the best-performing models with our GLMHA. Our results show up to a 7.7 Giga FLOPs reduction with 370K fewer parameters required to closely retain the original performance of the best-performing models that employ CSA.
Revisiting Essential and Nonessential Settings of Evidential Deep Learning
Venue: arXiv cs.LG 2024
Year: 2024

Evidential Deep Learning (EDL) is an emerging method for uncertainty estimation that provides reliable predictive uncertainty in a single forward pass, attracting significant attention. Grounded in subjective logic, EDL derives Dirichlet concentration parameters from neural networks to construct a Dirichlet probability density function (PDF), modeling the distribution of class probabilities. Despite its success, EDL incorporates several nonessential settings: In model construction, (1) a commonly ignored prior weight parameter is fixed to the number of classes, while its value actually impacts the balance between the proportion of evidence and its magnitude in deriving predictive scores. In model optimization, (2) the empirical risk features a variance-minimizing optimization term that biases the PDF towards a Dirac delta function, potentially exacerbating overconfidence. (3) Additionally, the structural risk typically includes a KL-divergence-minimizing regularization, whose optimization direction extends beyond the intended purpose and contradicts common sense, diminishing the information carried by the evidence magnitude. Therefore, we propose Re-EDL, a simplified yet more effective variant of EDL, by relaxing the nonessential settings and retaining the essential one, namely, the adoption of projected probability from subjective logic. Specifically, Re-EDL treats the prior weight as an adjustable hyperparameter rather than a fixed scalar, and directly optimizes the expectation of the Dirichlet PDF provided by deprecating both the variance-minimizing optimization term and the divergence regularization term. Extensive experiments and state-of-the-art performance validate the effectiveness of our method. The source code is available at https://github.com/MengyuanChen21/Re-EDL.
Analyzing Mass School Shootings in the United States from 1999 to 2024 with Game Theory, Probability Analysis, and Machine Learning
Venue: arXiv cs.CY 2024
Year: 2024

Public safety is vital to every country, especially school safety. In the United States, students and educators are concerned about school shootings. There are critical needs to understand the patterns of school shootings. Without this understanding, we cannot take action to prevent school shootings. Existing research that includes statistical analysis usually focuses on public mass shootings or just shooting incidents that have occurred in the past and there are hardly any articles focusing on mass school shootings. Here we firstly define mathematic models through gam theory. Then, we evaluate shootings events in schools for recently 26-year (1999-2024). Compared with the number of mass school shootings in COVID-19 period, we predict the number of mass school shooting events in the US will be reduced through four machine learning models. We also identify that mass school shootings usually take average 31 minutes with four periods. The annual probability of mass school shootings is 1.23 E-5 (or one in 81,604) per school. The shootings mostly occur inside buildings, especially classrooms and hallways. By interpreting these data and conducting various statistical analysis, this will ultimately help the law enforcement and schools to reduce the future school shootings. The research data sets could be downloaded via the website: https://publicsafetyinfo.com
LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
Venue: arXiv cs.DC 2024
Year: 2024

The expanding context windows in large language models (LLMs) have greatly enhanced their capabilities in various applications, but they also introduce significant challenges in maintaining low latency, particularly in Time to First Token (TTFT). This paper identifies that the sharp rise in TTFT as context length increases is predominantly driven by queuing delays, which are caused by the growing demands for GPU Key-Value (KV) cache allocation clashing with the limited availability of KV cache blocks. To address this issue, we propose LayerKV, a simple yet effective plug-in method that effectively reduces TTFT without requiring additional hardware or compromising output performance, while seamlessly integrating with existing parallelism strategies and scheduling techniques. Specifically, LayerKV introduces layer-wise KV block allocation, management, and offloading for fine-grained control over system memory, coupled with an SLO-aware scheduler to optimize overall Service Level Objectives (SLOs). Comprehensive evaluations on representative models, ranging from 7B to 70B parameters, across various GPU configurations, demonstrate that LayerKV improves TTFT latency up to 69x and reduces SLO violation rates by 28.7%, significantly enhancing the user experience.
A Scheduling-Aware Defense Against Prefetching-Based Side-Channel Attacks
Venue: arXiv cs.CR 2024
Year: 2024

Modern computer processors use microarchitectural optimization mechanisms to improve performance. As a downside, such optimizations are prone to introducing side-channel vulnerabilities. Speculative loading of memory, called prefetching, is common in real-world CPUs and may cause such side-channel vulnerabilities: Prior work has shown that it can be exploited to bypass process isolation and leak secrets, such as keys used in RSA, AES, and ECDH implementations. However, to this date, no effective and efficient countermeasure has been presented that secures software on systems with affected prefetchers. In this work, we answer the question: How can a process defend against prefetch-based side channels? We first systematize prefetching-based side-channel vulnerabilities presented in academic literature so far. Next, we design and implement PreFence, a scheduling-aware defense against these side channels that allows processes to disable the prefetcher temporarily during security-critical operations. We implement our countermeasure for an x86_64 and an ARM processor; it can be adapted to any platform that allows to disable the prefetcher. We evaluate our defense and find that our solution reliably stops prefetch leakage. Our countermeasure causes negligible performance impact while no security-relevant code is executed, and its worst case performance is comparable to completely turning off the prefetcher. The expected average performance impact depends on the security-relevant code in the application and can be negligible as we demonstrate with a simple web server application. We expect our countermeasure could widely be integrated in commodity OS, and even be extended to signal generally security-relevant code to the kernel to allow coordinated application of countermeasures.
Distributed Monitoring of Timed Properties
Venue: arXiv cs.SE 2024
Year: 2024

In formal verification, runtime monitoring consists of observing the execution of a system in order to decide as quickly as possible whether or not it satisfies a given property. We consider monitoring in a distributed setting, for properties given as reachability timed automata. In such a setting, the system is made of several components, each equipped with its own local clock and monitor. The monitors observe events occurring on their associated component, and receive timestamped events from other monitors through FIFO channels. Since clocks are local, they cannot be perfectly synchronized, resulting in imprecise timestamps. Consequently, they must be seen as intervals, leading monitors to consider possible reorderings of events. In this context, each monitor aims to provide, as early as possible, a verdict on the property it is monitoring, based on its potentially incomplete and imprecise knowledge of the current execution. In this paper, we propose an on-line monitoring algorithm for timed properties, robust to time imprecision and partial information from distant components. We first identify the date at which a monitor can safely compute a verdict based on received events. We then propose a monitoring algorithm that updates this date when new information arrives, maintains the current set of states in which the property can reside, and updates its verdict accordingly.
Deep Multimodal Fusion for Semantic Segmentation of Remote Sensing Earth Observation Data
Venue: arXiv cs.CV 2024
Year: 2024

Accurate semantic segmentation of remote sensing imagery is critical for various Earth observation applications, such as land cover mapping, urban planning, and environmental monitoring. However, individual data sources often present limitations for this task. Very High Resolution (VHR) aerial imagery provides rich spatial details but cannot capture temporal information about land cover changes. Conversely, Satellite Image Time Series (SITS) capture temporal dynamics, such as seasonal variations in vegetation, but with limited spatial resolution, making it difficult to distinguish fine-scale objects. This paper proposes a late fusion deep learning model (LF-DLM) for semantic segmentation that leverages the complementary strengths of both VHR aerial imagery and SITS. The proposed model consists of two independent deep learning branches. One branch integrates detailed textures from aerial imagery captured by UNetFormer with a Multi-Axis Vision Transformer (MaxViT) backbone. The other branch captures complex spatio-temporal dynamics from the Sentinel-2 satellite image time series using a U-Net with Temporal Attention Encoder (U-TAE). This approach leads to state-of-the-art results on the FLAIR dataset, a large-scale benchmark for land cover segmentation using multi-source optical imagery. The findings highlight the importance of multi-modality fusion in improving the accuracy and robustness of semantic segmentation in remote sensing applications.
Drone Stereo Vision for Radiata Pine Branch Detection and Distance Measurement: Utilizing Deep Learning and YOLO Integration
Venue: arXiv cs.CV 2024
Year: 2024

This research focuses on the development of a drone equipped with pruning tools and a stereo vision camera to accurately detect and measure the spatial positions of tree branches. YOLO is employed for branch segmentation, while two depth estimation approaches, monocular and stereo, are investigated. In comparison to SGBM, deep learning techniques produce more refined and accurate depth maps. In the absence of ground-truth data, a fine-tuning process using deep neural networks is applied to approximate optimal depth values. This methodology facilitates precise branch detection and distance measurement, addressing critical challenges in the automation of pruning operations. The results demonstrate notable advancements in both accuracy and efficiency, underscoring the potential of deep learning to drive innovation and enhance automation in the agricultural sector.
Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents
Venue: arXiv cs.CL 2024
Year: 2024

Instructional documents are rich sources of knowledge for completing various tasks, yet their unique challenges in conversational question answering (CQA) have not been thoroughly explored. Existing benchmarks have primarily focused on basic factual question-answering from single narrative documents, making them inadequate for assessing a model`s ability to comprehend complex real-world instructional documents and provide accurate step-by-step guidance in daily life. To bridge this gap, we present InsCoQA, a novel benchmark tailored for evaluating large language models (LLMs) in the context of CQA with instructional documents. Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents, reflecting the intricate and multi-faceted nature of real-world instructional tasks. Additionally, to comprehensively assess state-of-the-art LLMs on the InsCoQA benchmark, we propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.
Pointwise order of generalized Hofstadter functions $G, H$ and beyond
Venue: arXiv cs.DM 2024
Year: 2024

Hofstadter's $G$ function is recursively defined via $G(0)=0$ and then $G(n)=n-G(G(n-1))$. Following Hofstadter, a family $(F_k)$ of similar functions is obtained by varying the number $k$ of nested recursive calls in this equation. We establish here that this family is ordered pointwise: for all $k$ and $n$, $F_k(n) \le F_{k+1}(n)$. For achieving this, a detour is made via infinite morphic words generalizing the Fibonacci word. Various properties of these words are proved, concerning the lengths of substituted prefixes of these words and the counts of some specific letters in these prefixes. We also relate the limits of $\frac{1}{n}F_k(n)$ to the frequencies of letters in the considered words.
Asymmetrically connected reservoir networks learn better
Venue: arXiv cs.NE 2024
Year: 2024

We show that connectivity within the high-dimensional recurrent layer of a reservoir network is crucial for its performance. To this end, we systematically investigate the impact of network connectivity on its performance, i.e., we examine the symmetry and structure of the reservoir in relation to its computational power. Reservoirs with random and asymmetric connections are found to perform better for an exemplary Mackey-Glass time series than all structured reservoirs, including biologically inspired connectivities, such as small-world topologies. This result is quantified by the information processing capacity of the different network topologies which becomes highest for asymmetric and randomly connected networks.
Measurement challenges in AI catastrophic risk governance and safety frameworks
Venue: arXiv cs.CY 2024
Year: 2024

Safety frameworks represent a significant development in AI governance: they are the first type of publicly shared catastrophic risk management framework developed by major AI companies and focus specifically on AI scaling decisions. I identify six critical measurement challenges in their implementation and propose three policy recommendations to improve their validity and reliability.
A Reconfigurable Approximate Computing RISC-V Platform for Fault-Tolerant Applications
Venue: arXiv cs.AR 2024
Year: 2024

The demand for energy-efficient and high performance embedded systems drives the evolution of new hardware architectures, including concepts like approximate computing. This paper presents a novel reconfigurable embedded platform named "phoeniX", using the standard RISC-V ISA, maximizing energy efficiency while maintaining acceptable application-level accuracy. The platform enables the integration of approximate circuits at the core level with diverse structures, accuracies, and timings without requiring modifications to the core, particularly in the control logic. The platform introduces novel control features, allowing configurable trade-offs between accuracy and energy consumption based on specific application requirements. To evaluate the effectiveness of the platform, experiments were conducted on a set of applications, such as image processing and Dhrystone benchmark. The core with its original execution engine, occupies 0.024mm2 of area, with average power consumption of 4.23mW at 1.1V operating voltage, average energy-efficiency of 7.85pJ per operation at 620MHz frequency in 45nm CMOS technology. The configurable platform with a highly optimized 3-stage pipelined RV32I(E)M architecture, possesses a DMIPS/MHz of 1.89, and a CPI of 1.13, showcasing remarkable capabilities for an embedded processor.