OpenRaiser/Intern-Atlas huggingface dataset:
VisiTherS: Visible-thermal infrared stereo disparity estimation of human silhouette
Venue: arXiv cs.CV 2023
Year: 2023

This paper presents a novel approach for visible-thermal infrared stereoscopy, focusing on the estimation of disparities of human silhouettes. Visible-thermal infrared stereo poses several challenges, including occlusions and differently textured matching regions in both spectra. Finding matches between two spectra with varying colors, textures, and shapes adds further complexity to the task. To address the aforementioned challenges, this paper proposes a novel approach where a high-resolution convolutional neural network is used to better capture relationships between the two spectra. To do so, a modified HRNet backbone is used for feature extraction. This HRNet backbone is capable of capturing fine details and textures as it extracts features at multiple scales, thereby enabling the utilization of both local and global information. For matching visible and thermal infrared regions, our method extracts features on each patch using two modified HRNet streams. Features from the two streams are then combined for predicting the disparities by concatenation and correlation. Results on public datasets demonstrate the effectiveness of the proposed approach by improving the results by approximately 18 percentage points on the $\leq$ 1 pixel error, highlighting its potential for improving accuracy in this task. The code of VisiTherS is available on GitHub at the following link https://github.com/philippeDG/VisiTherS.
An Investigation of Face Mask Use with Busking Videos on YouTube during COVID-19: a Case Study in South Korea
Venue: arXiv cs.CY 2023
Year: 2023

Wearing face mask is an effective measure to reduce the risk of COVID-19 infections and control its transmission, thus its usage survey is important for better policy decision to mitigate the epidemic spread. Current existing worldwide surveys are mostly self-reported, whose accuracies are hard to guaranteed, and may exaggerate the percentage of face mask wearing. Therefore, we collected busking videos with a large amount on YouTube from December 2019 to December 2020, mainly from South Korea, and reported an objective investigation of face mask use in the crowds outdoor. It is found that the face mask wearing rate has an obvious positive correlation with effective reproductive number (Rt) in the South Korea, which indicates that the people in South Korea kept sensitive to the COVID-19 epidemic. The face mask wearing rate in South Korea is higher than some other countries, and two rate droppings in June and September also corresponds to the temporary remission in 2020. This study shows significant potentials to utilize public big video data to make an accurate worldwide survey of face mask use with the support of deep learning technology.
Input Augmentation with SAM: Boosting Medical Image Segmentation with Segmentation Foundation Model
Venue: arXiv cs.CV 2023
Year: 2023

The Segment Anything Model (SAM) is a recently developed large model for general-purpose segmentation for computer vision tasks. SAM was trained using 11 million images with over 1 billion masks and can produce segmentation results for a wide range of objects in natural scene images. SAM can be viewed as a general perception model for segmentation (partitioning images into semantically meaningful regions). Thus, how to utilize such a large foundation model for medical image segmentation is an emerging research target. This paper shows that although SAM does not immediately give high-quality segmentation for medical image data, its generated masks, features, and stability scores are useful for building and training better medical image segmentation models. In particular, we demonstrate how to use SAM to augment image input for commonly-used medical image segmentation models (e.g., U-Net). Experiments on three segmentation tasks show the effectiveness of our proposed SAMAug method. The code is available at \url{https://github.com/yizhezhang2000/SAMAug}.
Differentially Private Synthetic Data Generation via Lipschitz-Regularised Variational Autoencoders
Venue: arXiv cs.LG 2023
Year: 2023

Synthetic data has been hailed as the silver bullet for privacy preserving data analysis. If a record is not real, then how could it violate a person's privacy? In addition, deep-learning based generative models are employed successfully to approximate complex high-dimensional distributions from data and draw realistic samples from this learned distribution. It is often overlooked though that generative models are prone to memorising many details of individual training records and often generate synthetic data that too closely resembles the underlying sensitive training data, hence violating strong privacy regulations as, e.g., encountered in health care. Differential privacy is the well-known state-of-the-art framework for guaranteeing protection of sensitive individuals' data, allowing aggregate statistics and even machine learning models to be released publicly without compromising privacy. The training mechanisms however often add too much noise during the training process, and thus severely compromise the utility of these private models. Even worse, the tight privacy budgets do not allow for many training epochs so that model quality cannot be properly controlled in practice. In this paper we explore an alternative approach for privately generating data that makes direct use of the inherent stochasticity in generative models, e.g., variational autoencoders. The main idea is to appropriately constrain the continuity modulus of the deep models instead of adding another noise mechanism on top. For this approach, we derive mathematically rigorous privacy guarantees and illustrate its effectiveness with practical experiments.
A Deep Neural Network Deployment Based on Resistive Memory Accelerator Simulation
Venue: arXiv cs.AR 2023
Year: 2023

The objective of this study is to illustrate the process of training a Deep Neural Network (DNN) within a Resistive RAM (ReRAM) Crossbar-based simulation environment using CrossSim, an Application Programming Interface (API) developed for this purpose. The CrossSim API is designed to simulate neural networks while taking into account factors that may affect the accuracy of solutions during training on non-linear and noisy ReRAM devices. ReRAM-based neural cores that serve as memory accelerators for digital cores on a chip can significantly reduce energy consumption by minimizing data transfers between the processor and SRAM and DRAM. CrossSim employs lookup tables obtained from experimentally derived datasets of real fabricated ReRAM devices to digitally reproduce noisy weight updates to the neural network. The CrossSim directory comprises eight device configurations that operate at different temperatures and are made of various materials. This study aims to analyse the results of training a Neural Network on the Breast Cancer Wisconsin (Diagnostic) dataset using CrossSim, plotting the innercore weight updates and average training and validation loss to investigate the outcomes of all the devices.
Unsupervised CD in satellite image time series by contrastive learning and feature tracking
Venue: arXiv cs.CV 2023
Year: 2023

While unsupervised change detection using contrastive learning has been significantly improved the performance of literature techniques, at present, it only focuses on the bi-temporal change detection scenario. Previous state-of-the-art models for image time-series change detection often use features obtained by learning for clustering or training a model from scratch using pseudo labels tailored to each scene. However, these approaches fail to exploit the spatial-temporal information of image time-series or generalize to unseen scenarios. In this work, we propose a two-stage approach to unsupervised change detection in satellite image time-series using contrastive learning with feature tracking. By deriving pseudo labels from pre-trained models and using feature tracking to propagate them among the image time-series, we improve the consistency of our pseudo labels and address the challenges of seasonal changes in long-term remote sensing image time-series. We adopt the self-training algorithm with ConvLSTM on the obtained pseudo labels, where we first use supervised contrastive loss and contrastive random walks to further improve the feature correspondence in space-time. Then a fully connected layer is fine-tuned on the pre-trained multi-temporal features for generating the final change maps. Through comprehensive experiments on two datasets, we demonstrate consistent improvements in accuracy on fitting and inference scenarios.
Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning
Venue: arXiv cs.SE 2023
Year: 2023

Code comment generation aims at generating natural language descriptions for a code snippet to facilitate developers' program comprehension activities. Despite being studied for a long time, a bottleneck for existing approaches is that given a code snippet, they can only generate one comment while developers usually need to know information from diverse perspectives such as what is the functionality of this code snippet and how to use it. To tackle this limitation, this study empirically investigates the feasibility of utilizing large language models (LLMs) to generate comments that can fulfill developers' diverse intents. Our intuition is based on the facts that (1) the code and its pairwise comment are used during the pre-training process of LLMs to build the semantic connection between the natural language and programming language, and (2) comments in the real-world projects, which are collected for the pre-training, usually contain different developers' intents. We thus postulate that the LLMs can already understand the code from different perspectives after the pre-training. Indeed, experiments on two large-scale datasets demonstrate the rationale of our insights: by adopting the in-context learning paradigm and giving adequate prompts to the LLM (e.g., providing it with ten or more examples), the LLM can significantly outperform a state-of-the-art supervised learning approach on generating comments with multiple intents. Results also show that customized strategies for constructing the prompts and post-processing strategies for reranking the results can both boost the LLM's performances, which shed light on future research directions for using LLMs to achieve comment generation.
Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens
Venue: arXiv cs.CL 2023
Year: 2023

Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consolidate these findings by evaluating surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, after which they begin to diverge from humanlike expectations. Additionally, newly-trained smaller model variants reveal a 'tipping point' at convergence, after which the decrease in language model perplexity begins to result in poorer fits to human reading times. These results suggest that the massive amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger pre-trained language models, and that a certain degree of model capacity is necessary for Transformer-based language models to capture humanlike expectations.
Three-dimensional hand guidance by midair haptic display
Venue: arXiv cs.HC 2023
Year: 2023

Guiding human movements using tactile information is one of the promising applications of haptics. Using midair ultrasonic haptic stimulation, it is possible to guide a hand without visual information.However, the information of movement shown by conventional methods was partial. It has not been shown a method to guide a hand to an arbitrary point in three dimensional space. In this study, we propose a method of guiding the hand to the top of a virtual cone presented haptically and evaluate the effectiveness of the method through experiments. As a result, the method guided the participant's hand to the goal in a 30 cm cube workspace with an error of 64.34 mm
Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
Venue: arXiv cs.DC 2023
Year: 2023

The Mixture of Experts (MoE) model becomes an important choice of large language models nowadays because of its scalability with sublinear computational complexity for training and inference. However, existing MoE models suffer from two critical drawbacks, 1) tremendous inner-node and inter-node communication overhead introduced by all-to-all dispatching and gathering, and 2) limited scalability for the backbone because of the bound data parallel and expert parallel to scale in the expert dimension. In this paper, we systematically analyze these drawbacks in terms of training efficiency in the parallel framework view and propose a novel MoE architecture called Pipeline MoE (PPMoE) to tackle them. PPMoE builds expert parallel incorporating with tensor parallel and replaces communication-intensive all-to-all dispatching and gathering with a simple tensor index slicing and inner-node all-reduce. Besides, it is convenient for PPMoE to integrate pipeline parallel to further scale the backbone due to its flexible parallel architecture. Extensive experiments show that PPMoE not only achieves a more than $1.75\times$ speed up compared to existing MoE architectures but also reaches $90\%$ throughput of its corresponding backbone model that is $20\times$ smaller.
STNet: Spatial and Temporal feature fusion network for change detection in remote sensing images
Venue: arXiv cs.CV 2023
Year: 2023

As an important task in remote sensing image analysis, remote sensing change detection (RSCD) aims to identify changes of interest in a region from spatially co-registered multi-temporal remote sensing images, so as to monitor the local development. Existing RSCD methods usually formulate RSCD as a binary classification task, representing changes of interest by merely feature concatenation or feature subtraction and recovering the spatial details via densely connected change representations, whose performances need further improvement. In this paper, we propose STNet, a RSCD network based on spatial and temporal feature fusions. Specifically, we design a temporal feature fusion (TFF) module to combine bi-temporal features using a cross-temporal gating mechanism for emphasizing changes of interest; a spatial feature fusion module is deployed to capture fine-grained information using a cross-scale attention mechanism for recovering the spatial details of change representations. Experimental results on three benchmark datasets for RSCD demonstrate that the proposed method achieves the state-of-the-art performance. Code is available at https://github.com/xwmaxwma/rschange.
Pred-NBV: Prediction-guided Next-Best-View for 3D Object Reconstruction
Venue: arXiv cs.RO 2023
Year: 2023

Prediction-based active perception has shown the potential to improve the navigation efficiency and safety of the robot by anticipating the uncertainty in the unknown environment. The existing works for 3D shape prediction make an implicit assumption about the partial observations and therefore cannot be used for real-world planning and do not consider the control effort for next-best-view planning. We present Pred-NBV, a realistic object shape reconstruction method consisting of PoinTr-C, an enhanced 3D prediction model trained on the ShapeNet dataset, and an information and control effort-based next-best-view method to address these issues. Pred-NBV shows an improvement of 25.46% in object coverage over the traditional methods in the AirSim simulator, and performs better shape completion than PoinTr, the state-of-the-art shape completion model, even on real data obtained from a Velodyne 3D LiDAR mounted on DJI M600 Pro.
Collie: Finding Performance Anomalies in RDMA Subsystems
Venue: arXiv cs.NI 2023
Year: 2023

High-speed RDMA networks are getting rapidly adopted in the industry for their low latency and reduced CPU overheads. To verify that RDMA can be used in production, system administrators need to understand the set of application workloads that can potentially trigger abnormal performance behaviors (e.g., unexpected low throughput, PFC pause frame storm). We design and implement Collie, a tool for users to systematically uncover performance anomalies in RDMA subsystems without the need to access hardware internal designs. Instead of individually testing each hardware device (e.g., NIC, memory, PCIe), Collie is holistic, constructing a comprehensive search space for application workloads. Collie then uses simulated annealing to drive RDMA-related performance and diagnostic counters to extreme value regions to find workloads that can trigger performance anomalies. We evaluate Collie on combinations of various RDMA NIC, CPU, and other hardware components. Collie found 15 new performance anomalies. All of them are acknowledged by the hardware vendors. 7 of them are already fixed after we reported them. We also present our experience in using Collie to avoid performance anomalies for an RDMA RPC library and an RDMA distributed machine learning framework.
The Logic of Prefixes and Suffixes is Elementary under Homogeneity
Venue: arXiv cs.LO 2023
Year: 2023

In this paper, we study the finite satisfiability problem for the logic BE under the homogeneity assumption. BE is the cornerstone of Halpern and Shoham's interval temporal logic, and features modal operators corresponding to the prefix (a.k.a. "Begins") and suffix (a.k.a. "Ends") relations on intervals. In terms of complexity, BE lies in between the "Chop logic C", whose satisfiability problem is known to be non-elementary, and the PSPACE-complete interval logic D of the sub-interval (a.k.a. "During") relation. BE was shown to be EXPSPACE-hard, and the only known satisfiability procedure is primitive recursive, but not elementary. Our contribution consists of tightening the complexity bounds of the satisfiability problem for BE, by proving it to be EXPSPACE-complete. We do so by devising an equi-satisfiable normal form with boundedly many nested modalities. The normalization technique resembles Scott's quantifier elimination, but it turns out to be much more involved due to the limitations enforced by the homogeneity assumption.
Machine learning framework for end-to-end implementation of Incident duration prediction
Venue: arXiv cs.LG 2023
Year: 2023

Traffic congestion caused by non-recurring incidents such as vehicle crashes and debris is a key issue for Traffic Management Centers (TMCs). Clearing incidents in a timely manner is essential for improving safety and reducing delays and emissions for the traveling public. However, TMCs and other responders face a challenge in predicting the duration of incidents (until the roadway is clear), making decisions of what resources to deploy difficult. To address this problem, this research developed an analytical framework and end-to-end machine-learning solution for predicting incident duration based on information available as soon as an incident report is received. Quality predictions of incident duration can help TMCs and other responders take a proactive approach in deploying responder services such as tow trucks, maintenance crews or activating alternative routes. The predictions use a combination of classification and regression machine learning modules. The performance of the developed solution has been evaluated based on the Mean Absolute Error (MAE), or deviation from the actual incident duration as well as Area Under the Curve (AUC) and Mean Absolute Percentage Error (MAPE). The results showed that the framework significantly improved incident duration prediction compared to methods from previous research.
Joint Beamforming and Phase Shift Design for Hybrid-IRS-and-UAV-aided Directional Modulation Network
Venue: arXiv cs.IT 2023
Year: 2023

Recently, intelligent reflecting surface (IRS) and unmanned aerial vehicle (UAV) have been introduced into wireless communication systems to enhance the performance of air-ground transmission. To make a good balance between performance, cost, and power consumption, a hybrid-IRS-and-UAV-assisted directional modulation (DM) network is investigated in this paper, where the hybrid IRS consists of passive and active reflecting elements. To maximize the achievable rate, three optimization algorithms, called maximum signal-to-noise ratio (SNR)-fractional programming (FP) (Max-SNR-FP), maximum SNR-equal amplitude reflecting (EAR) (Max-SNR-EAR), and maximum SNR-majorization-minimization (MM) (Max-SNR-MM), are proposed to jointly design the beamforming vector and phase shift matrix (PSM) of hybrid IRS by alternately optimizing one and giving another. The Max-SNR-FP method employs the successive convex approximation and FP methods to derive the beamforming vector and hybrid IRS PSM. The Max-SNR-EAR method adopts the maximum signal-to-leakage-noise ratio method and the criteria of phase alignment and EAR to design them. In addition, the Max-SNR-MM method utilizes the MM criterion to derive the IRS PSM. Simulation results show that the rates harvested by the proposed three methods are slightly lower than those of active IRS with higher power consumption, which are 35 percent higher than those of no IRS and random phase IRS, while passive IRS achieves only about 17 percent rate gain over the latter. Moreover, compared to Max-SNR-FP, the proposed Max-SNR-EAR and Max-SNR-MM methods make an obvious complexity degradation at the price of a slight performance loss.
LayerNAS: Neural Architecture Search in Polynomial Complexity
Venue: arXiv cs.LG 2023
Year: 2023

Neural Architecture Search (NAS) has become a popular method for discovering effective model architectures, especially for target hardware. As such, NAS methods that find optimal architectures under constraints are essential. In our paper, we propose LayerNAS to address the challenge of multi-objective NAS by transforming it into a combinatorial optimization problem, which effectively constrains the search complexity to be polynomial. For a model architecture with $L$ layers, we perform layerwise-search for each layer, selecting from a set of search options $\mathbb{S}$. LayerNAS groups model candidates based on one objective, such as model size or latency, and searches for the optimal model based on another objective, thereby splitting the cost and reward elements of the search. This approach limits the search complexity to $ O(H \cdot |\mathbb{S}| \cdot L) $, where $H$ is a constant set in LayerNAS. Our experiments show that LayerNAS is able to consistently discover superior models across a variety of search spaces in comparison to strong baselines, including search spaces derived from NATS-Bench, MobileNetV2 and MobileNetV3.