OpenRaiser/Intern-Atlas huggingface dataset:
SVNR: Spatially-variant Noise Removal with Denoising Diffusion
Venue: arXiv cs.CV 2023
Year: 2023

Denoising diffusion models have recently shown impressive results in generative tasks. By learning powerful priors from huge collections of training images, such models are able to gradually modify complete noise to a clean natural image via a sequence of small denoising steps, seemingly making them well-suited for single image denoising. However, effectively applying denoising diffusion models to removal of realistic noise is more challenging than it may seem, since their formulation is based on additive white Gaussian noise, unlike noise in real-world images. In this work, we present SVNR, a novel formulation of denoising diffusion that assumes a more realistic, spatially-variant noise model. SVNR enables using the noisy input image as the starting point for the denoising diffusion process, in addition to conditioning the process on it. To this end, we adapt the diffusion process to allow each pixel to have its own time embedding, and propose training and inference schemes that support spatially-varying time maps. Our formulation also accounts for the correlation that exists between the condition image and the samples along the modified diffusion process. In our experiments we demonstrate the advantages of our approach over a strong diffusion model baseline, as well as over a state-of-the-art single image denoising method.
A serial dual-channel library occupancy detection system based on Faster RCNN
Venue: arXiv cs.CV 2023
Year: 2023

The phenomenon of seat occupancy in university libraries is a prevalent issue. However, existing solutions, such as software-based seat reservations and sensors-based occupancy detection, have proven to be inadequate in effectively addressing this problem. In this study, we propose a novel approach: a serial dual-channel object detection model based on Faster RCNN. This model is designed to discern all instances of occupied seats within the library and continuously update real-time information regarding seat occupancy status. To train the neural network, a distinctive dataset is utilized, which blends virtual images generated using Unreal Engine 5 (UE5) with real-world images. Notably, our test results underscore the remarkable performance uplift attained through the application of self-generated virtual datasets in training Convolutional Neural Networks (CNNs), particularly within specialized scenarios. Furthermore, this study introduces a pioneering detection model that seamlessly amalgamates the Faster R-CNN-based object detection framework with a transfer learning-based object classification algorithm. This amalgamation not only significantly curtails the computational resources and time investments needed for neural network training but also considerably heightens the efficiency of single-frame detection rates. Additionally, a user-friendly web interface and a mobile application have been meticulously developed, constituting a computer vision-driven platform for detecting seat occupancy within library premises. Noteworthy is the substantial enhancement in seat occupancy recognition accuracy, coupled with a reduction in computational resources required for neural network training, collectively contributing to a considerable amplification in the overall efficiency of library seat management.
Retrospective: Flipping Bits in Memory Without Accessing Them: An Experimental Study of DRAM Disturbance Errors
Venue: arXiv cs.CR 2023
Year: 2023

Our ISCA 2014 paper provided the first scientific and detailed characterization, analysis, and real-system demonstration of what is now popularly known as the RowHammer phenomenon (or vulnerability) in modern commodity DRAM chips, which are used as main memory in almost all modern computing systems. It experimentally demonstrated that more than 80% of all DRAM modules we tested from the three major DRAM vendors were vulnerable to the RowHammer read disturbance phenomenon: one can predictably induce bitflips (i.e., data corruption) in real DRAM modules by repeatedly accessing a DRAM row and thus causing electrical disturbance to physically nearby rows. We showed that a simple unprivileged user-level program induced RowHammer bitflips in multiple real systems and suggested that a security attack can be built using this proof-of-concept to hijack control of the system or cause other harm. To solve the RowHammer problem, our paper examined seven different approaches (including a novel probabilistic approach that has very low cost), some of which influenced or were adopted in different industrial products. Many later works from various research communities examined RowHammer, building real security attacks, proposing new defenses, further analyzing the problem at various (e.g., device/circuit, architecture, and system) levels, and exploiting RowHammer for various purposes (e.g., to reverse-engineer DRAM chips). Industry has worked to mitigate the problem, changing both memory controllers and DRAM standards/chips. Two major DRAM vendors finally wrote papers on the topic in 2023, describing their current approaches to mitigate RowHammer. Research & development on RowHammer in both academia & industry continues to be very active and fascinating. This short retrospective provides a brief analysis of our ISCA 2014 paper and its impact.
Blockchain Nodes are Heterogeneous and Your P2P Overlay Should be Too: PODS
Venue: arXiv cs.DC 2023
Year: 2023

At the core of each blockchain system, parties communicate through a peer-to-peer (P2P) overlay. Unfortunately, recent evidence suggests these P2P overlays represent a significant bottleneck for transaction throughput and scalability. Furthermore, they enable a number of attacks. We argue that these performance and security problems arise because current P2P overlays cannot fully capture the complexity of a blockchain system as they do not offer flexibility to accommodate node heterogeneity. We propose a novel approach to address these issues: P2P Overlay Domains with Sovereignty (PODS), which allows nodes in a single overlay to belong to multiple heterogeneous groups, called domains. Each domain features its own set of protocols, tailored to the characteristics and needs of its nodes. To demonstrate the effectiveness of PODS, we design and implement two novel node discovery protocols: FedKad and SovKad. Using a custom simulator, we show that node discovery using PODS (SovKad) architecture outperforms both single overlay (Kademlia) and multi-overlay (FedKad) architectures in terms of hop count and success rate, though FedKad requires slightly less bandwidth.
Pseudo-Bag Mixup Augmentation for Multiple Instance Learning-Based Whole Slide Image Classification
Venue: arXiv cs.CV 2023
Year: 2023

Given the special situation of modeling gigapixel images, multiple instance learning (MIL) has become one of the most important frameworks for Whole Slide Image (WSI) classification. In current practice, most MIL networks often face two unavoidable problems in training: i) insufficient WSI data and ii) the sample memorization inclination inherent in neural networks. These problems may hinder MIL models from adequate and efficient training, suppressing the continuous performance promotion of classification models on WSIs. Inspired by the basic idea of Mixup, this paper proposes a new Pseudo-bag Mixup (PseMix) data augmentation scheme to improve the training of MIL models. This scheme generalizes the Mixup strategy for general images to special WSIs via pseudo-bags so as to be applied in MIL-based WSI classification. Cooperated by pseudo-bags, our PseMix fulfills the critical size alignment and semantic alignment in Mixup strategy. Moreover, it is designed as an efficient and decoupled method, neither involving time-consuming operations nor relying on MIL model predictions. Comparative experiments and ablation studies are specially designed to evaluate the effectiveness and advantages of our PseMix. Experimental results show that PseMix could often assist state-of-the-art MIL networks to refresh their classification performance on WSIs. Besides, it could also boost the generalization performance of MIL models in special test scenarios, and promote their robustness to patch occlusion and label noise. Our source code is available at https://github.com/liupei101/PseMix.
Learning to Pan-sharpening with Memories of Spatial Details
Venue: arXiv cs.CV 2023
Year: 2023

Pan-sharpening, as one of the most commonly used techniques in remote sensing systems, aims to inject spatial details from panchromatic images into multispectral images (MS) to obtain high-resolution multispectral images. Since deep learning has received widespread attention because of its powerful fitting ability and efficient feature extraction, a variety of pan-sharpening methods have been proposed to achieve remarkable performance. However, current pan-sharpening methods usually require the paired panchromatic (PAN) and MS images as input, which limits their usage in some scenarios. To address this issue, in this paper we observe that the spatial details from PAN images are mainly high-frequency cues, i.e., the edges reflect the contour of input PAN images. This motivates us to develop a PAN-agnostic representation to store some base edges, so as to compose the contour for the corresponding PAN image via them. As a result, we can perform the pan-sharpening task with only the MS image when inference. To this end, a memory-based network is adapted to extract and memorize the spatial details during the training phase and is used to replace the process of obtaining spatial information from PAN images when inference, which is called Memory-based Spatial Details Network (MSDN). Finally, we integrate the proposed MSDN module into the existing deep learning-based pan-sharpening methods to achieve an end-to-end pan-sharpening network. With extensive experiments on the Gaofen1 and WorldView-4 satellites, we verify that our method constructs good spatial details without PAN images and achieves the best performance. The code is available at https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git.
Deterministic End-to-End Transmission to Optimize the Network Efficiency and Quality of Service: A Paradigm Shift in 6G
Venue: arXiv cs.NI 2023
Year: 2023

Toward end-to-end mobile service provision with optimized network efficiency and quality of service, tremendous efforts have been devoted in upgrading mobile applications, transport and internet networks, and wireless communication networks for many years. However, the inherent loose coordination between different layers in the end-to-end communication networks leads to unreliable data transmission with uncontrollable packet delay and packet error rate, and a terrible waste of network resources incurred for data re-transmission. In an attempt to shed some lights on how to tackle these challenges, design methodologies and some solutions for deterministic end-to-end transmission for 6G and beyond are presented, which will bring a paradigm shift to the end-to-end wireless communication networks.
Multi-IMU with Online Self-Consistency for Freehand 3D Ultrasound Reconstruction
Venue: arXiv cs.CV 2023
Year: 2023

Ultrasound (US) imaging is a popular tool in clinical diagnosis, offering safety, repeatability, and real-time capabilities. Freehand 3D US is a technique that provides a deeper understanding of scanned regions without increasing complexity. However, estimating elevation displacement and accumulation error remains challenging, making it difficult to infer the relative position using images alone. The addition of external lightweight sensors has been proposed to enhance reconstruction performance without adding complexity, which has been shown to be beneficial. We propose a novel online self-consistency network (OSCNet) using multiple inertial measurement units (IMUs) to improve reconstruction performance. OSCNet utilizes a modal-level self-supervised strategy to fuse multiple IMU information and reduce differences between reconstruction results obtained from each IMU data. Additionally, a sequence-level self-consistency strategy is proposed to improve the hierarchical consistency of prediction results among the scanning sequence and its sub-sequences. Experiments on large-scale arm and carotid datasets with multiple scanning tactics demonstrate that our OSCNet outperforms previous methods, achieving state-of-the-art reconstruction performance.
Inferring the Goals of Communicating Agents from Actions and Instructions
Venue: arXiv cs.AI 2023
Year: 2023

When humans cooperate, they frequently coordinate their activity through both verbal communication and non-verbal actions, using this information to infer a shared goal and plan. How can we model this inferential ability? In this paper, we introduce a model of a cooperative team where one agent, the principal, may communicate natural language instructions about their shared plan to another agent, the assistant, using GPT-3 as a likelihood function for instruction utterances. We then show how a third person observer can infer the team's goal via multi-modal Bayesian inverse planning from actions and instructions, computing the posterior distribution over goals under the assumption that agents will act and communicate rationally to achieve them. We evaluate this approach by comparing it with human goal inferences in a multi-agent gridworld, finding that our model's inferences closely correlate with human judgments (R = 0.96). When compared to inference from actions alone, we also find that instructions lead to more rapid and less uncertain goal inference, highlighting the importance of verbal communication for cooperative agents.
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model
Venue: arXiv cs.CV 2023
Year: 2023

Leveraging the extensive training data from SA-1B, the Segment Anything Model (SAM) demonstrates remarkable generalization and zero-shot capabilities. However, as a category-agnostic instance segmentation method, SAM heavily relies on prior manual guidance, including points, boxes, and coarse-grained masks. Furthermore, its performance in remote sensing image segmentation tasks remains largely unexplored and unproven. In this paper, we aim to develop an automated instance segmentation approach for remote sensing images, based on the foundational SAM model and incorporating semantic category information. Drawing inspiration from prompt learning, we propose a method to learn the generation of appropriate prompts for SAM. This enables SAM to produce semantically discernible segmentation results for remote sensing images, a concept we have termed RSPrompter. We also propose several ongoing derivatives for instance segmentation tasks, drawing on recent advancements within the SAM community, and compare their performance with RSPrompter. Extensive experimental results, derived from the WHU building, NWPU VHR-10, and SSDD datasets, validate the effectiveness of our proposed method. The code for our method is publicly available at kychen.me/RSPrompter.
A Review on Optimality Investigation Strategies for the Balanced Assignment Problem
Venue: arXiv cs.CC 2023
Year: 2023

Mathematical Selection is a method in which we select a particular choice from a set of such. It have always been an interesting field of study for mathematicians. Accordingly, Combinatorial Optimization is a sub field of this domain of Mathematical Selection, where we generally, deal with problems subjecting to Operation Research, Artificial Intelligence and many more promising domains. In a broader sense, an optimization problem entails maximising or minimising a real function by systematically selecting input values from within an allowed set and computing the function's value. A broad region of applied mathematics is the generalisation of metaheuristic theory and methods to other formulations. More broadly, optimization entails determining the finest virtues of some fitness function, offered a fixed space, which may include a variety of distinct types of decision variables and contexts. In this work, we will be working on the famous Balanced Assignment Problem, and will propose a comparative analysis on the Complexity Metrics of Computational Time for different Notions of solving the Balanced Assignment Problem.
Dual Identities Enabled Low-Latency Visual Networking for UAV Emergency Communication
Venue: arXiv cs.NI 2023
Year: 2023

The Unmanned Aerial Vehicle (UAV) swarm networks will play a crucial role in the B5G/6G network thanks to its appealing features, such as wide coverage and on-demand deployment. Emergency communication (EC) is essential to promptly inform UAVs of potential danger to avoid accidents, whereas the conventional communication-only feedback-based methods, which separate the digital and physical identities (DPI), bring intolerable latency and disturb the unintended receivers. In this paper, we present a novel DPI-Mapping solution to match the identities (IDs) of UAVs from dual domains for visual networking, which is the first solution that enables UAVs to communicate promptly with what they see without the tedious exchange of beacons. The IDs are distinguished dynamically by defining feature similarity, and the asymmetric IDs from different domains are matched via the proposed bio-inspired matching algorithm. We also consider Kalman filtering to combine the IDs and predict the states for accurate mapping. Experiment results show that the DPI-Mapping reduces individual inaccuracy of features and significantly outperforms the conventional broadcast-based and feedback-based methods in EC latency. Furthermore, it also reduces the disturbing messages without sacrificing the hit rate.
Representation Learning via Variational Bayesian Networks
Venue: arXiv cs.LG 2023
Year: 2023

We present Variational Bayesian Network (VBN) - a novel Bayesian entity representation learning model that utilizes hierarchical and relational side information and is particularly useful for modeling entities in the ``long-tail'', where the data is scarce. VBN provides better modeling for long-tail entities via two complementary mechanisms: First, VBN employs informative hierarchical priors that enable information propagation between entities sharing common ancestors. Additionally, VBN models explicit relations between entities that enforce complementary structure and consistency, guiding the learned representations towards a more meaningful arrangement in space. Second, VBN represents entities by densities (rather than vectors), hence modeling uncertainty that plays a complementary role in coping with data scarcity. Finally, we propose a scalable Variational Bayes optimization algorithm that enables fast approximate Bayesian inference. We evaluate the effectiveness of VBN on linguistic, recommendations, and medical inference tasks. Our findings show that VBN outperforms other existing methods across multiple datasets, and especially in the long-tail.
Theater Aid System for the Visually Impaired Through Transfer Learning of Spatio-Temporal Graph Convolution Networks
Venue: arXiv cs.CV 2023
Year: 2023

The aim of this research is to recognize human actions performed on stage to aid visually impaired and blind individuals. To achieve this, we have created a theatre human action recognition system that uses skeleton data captured by depth image as input. We collected new samples of human actions in a theatre environment, and then tested the transfer learning technique with three pre-trained Spatio-Temporal Graph Convolution Networks for skeleton-based human action recognition: the spatio-temporal graph convolution network, the two-stream adaptive graph convolution network, and the multi-scale disentangled unified graph convolution network. We selected the NTU-RGBD human action benchmark as the source domain and used our collected dataset as the target domain. We analyzed the transferability of the pre-trained models and proposed two configurations to apply and adapt the transfer learning technique to the diversity between the source and target domains. The use of transfer learning helped to improve the performance of the human action system within the context of theatre. The results indicate that Spatio-Temporal Graph Convolution Networks is positively transferred, and there was an improvement in performance compared to the baseline without transfer learning.
Why white noise is not enough? On using radio front-end models while designing 6G PHY
Venue: arXiv cs.NI 2023
Year: 2023

From generation to generation there are increasing requirements for wireless standards both in terms of spectral and energy efficiency. While up to now the layered wireless transceiver architecture worked allowing for, e.g., separation of channel decoding algorithms from front-end design, this may need reconsideration in the 6G era. Especially the hardware-originated distortions have to be taken into account while designing other layer algorithms as the high throughput and energy efficiency requirements will push these devices to their limit revealing their nonlinear characteristics. This position paper will shed some light on new degrees of freedom while cross-layer designing and controlling multicarrier and multiantenna transceivers of 6G systems.
Generalized Core Spanner Inexpressibility via Ehrenfeucht-Fraïssé Games for FC
Venue: arXiv cs.LO 2023
Year: 2023

Despite considerable research on document spanners, little is known about the expressive power of generalized core spanners. In this paper, we use Ehrenfeucht-Fraïssé games to obtain general inexpressibility lemmas for the logic FC (a finite-model variant of the theory of concatenation). Applying these lemmas give inexpressibility results for FC that we lift to generalized core spanners. In particular, we give several relations that cannot be selected by generalized core spanners, thus demonstrating the effectiveness of the inexpressibility lemmas. As an immediate consequence, we also gain new insights into the expressive power of core spanners.
Coding Estimation based on Rate Distortion Control of H.264 Encoded Videos for Low Latency Applications
Venue: arXiv cs.IT 2023
Year: 2023

In the field of video processing, advancements in video compression at various temporal and spatial resolutions which are needed in our research to quantify estimation of video quality whereabouts within spatial and temporal domain itself. It was necessary in our research to study the impacts of related video coding conditions upon perceptual quality due to issue raised by User Experience community regarding poor coding. But most of research studies are concerned with coding distortions affected by mostly due to poor coding which address high spatio-temporal resolutions. This paper presents overall 120 test scenarios for video sequences having low spatial and temporal spectral information were studied. Finally we concluded that even after achieving consistency within subjective scores, we got relevant data consistency after refining subjective scores, so it is not poor coding its due channel capacity which was traced out by rate distortion control.