Reviewerly/exHarmony huggingface dataset:
OblivIO: Securing Reactive Programs by Oblivious Execution with Bounded Traffic Overheads

Traffic analysis attacks remain a significant problem for online security. Communication between nodes can be observed by network level attackers as it inherently takes place in the open. Despite online services increasingly using encrypted traffic, the shape of the traffic is not hidden. To prevent traffic analysis, the shape of a system's traffic must be independent of secrets. We investigate adapting the data-oblivious approach the reactive setting and present OblivIO, a secure language for writing reactive programs driven by network events. Our approach pads with dummy messages to hide which program sends are genuinely executed. We use an information-flow type system to provably enforce timing-sensitive noninterference. The type system is extended with potentials to bound the overhead in traffic introduced by our approach. We address challenges that arise from joining data-oblivious and reactive programming and demonstrate the feasibility of our resulting language by developing an interpreter that implements security critical operations as constant-time algorithms.
FaceClone: Interactive Facial Shape and Motion Cloning System Using Multi-View Images

We present an interactive face cloning system that imitates users' facial expressions in virtual space. To do this, we construct a multi-view capture studio. Our system consists of offline and online stages. In the offline stage, facial shapes and light conditions are jointly optimized to obtain a realistic facial scan from static multi-view images. In the online stage, the facial scan is parameterized using landmarks and it is deformed according to the users' facial expressions. Furthermore, our system provides a GUI to allow user interaction with the generated facial clone. We demonstrate our system enables immersive communication in a virtual world.
Robust Safety for Move

A program that maintains key safety properties even when interacting with arbitrary untrusted code is said to enjoy robust safety. Proving that a program written in a mainstream language is robustly safe is typically challenging because it requires static verification tools that work precisely even in the presence of language features like dynamic dispatch and shared mutability. The emerging Move programming language was designed to support strong encapsulation and static verification in the service of secure smart contract programming. However, the language design has not been analysed using a theoretical framework like robust safety. In this paper, we define robust safety for the Move language and introduce a generic framework for static tools that wish to enforce it. Our framework consists of two abstract components: a program verifier that can prove an invariant holds in a closed-world setting (e.g., the Move Prover [16], [47]), and a novel encapsulator that checks if the verifier's result generalizes to an open-world setting. We formalise an escape analysis as an instantiation of the encapsulator and prove that it attains the required security properties. Finally, we implement our encapsulator as an extension to the Move Prover and use the combination to analyse a large representative benchmark set of real-world Move programs. This toolchain certifies >99% of the Move modules we analyse, validating that automatic enforcement of strong security properties like robust safety is practical for Move. Additionally, our results tell that security-centric language design can be effective in attaining strong security properties such as robust safety.
NARV: An Efficient Noise-Adaptive ResNet VAE for Joint Image Compression and Denoising

Image compression and denoising are two fundamental problems in image processing and have many real-world applications. In recent years, learning-based compression methods have achieved promising rate-distortion performance. However, most learning-based compression methods are trained with clean-to-clean image pairs in a noise-unaware manner, resulting in bits misallocation when input images contain noticeable noises. To tackle this problem, we propose an efficient end-to-end method, named Noise-Adaptive ResNet VAE (NARV), which is capable of handling both clean and noisy input images. Experimental results show that NARV achieves better rate-distortion performance under various noise levels while maintaining a small model size and fast processing time for practical implementation. The code of our method is publicly accessible at https://github.com/Eventhyn/NARV
Predicting Frags in Tactic Games at KnowledgePit.ai: ICME 2023 Grand Challenge Report

We describe a data science competition ICME 2023 Grand Challenge: Predicting Frags in Tactic Games that was organized in association with the IEEE ICME conference series at the KnowledgePit.ai platform. This challenge was the second in a series of competitions related to the analysis of data from a turn-based tactic video game Tactical Troops: Anthracite Shift. We discuss the competition's scope and significance of the considered research problem. We also overview the construction of the baseline solution and the most interesting results obtained by competing teams. We also indicate how the challenge outcomes fit into our future plans related to video game data analytics and the future applications of our KnowledgePit.ai platform.
STGA-Net: Spatial-Temporal Graph Attention Network for Skeleton-Based Temporal Action Segmentation

Temporal action segmentation aims at dense labeling of video frames with a series of action classes in long and untrimmed videos. However, previous methods heavily rely on generating an initial prediction with temporal convolutional layers and refining the predictions over the following stages based on RGB features. This results in the lack of an explicit action segment transition rule and loss of high-level semantic information, such as complex spatial-temporal correlation among the human joints between frames. Therefore, we present a spatial-temporal graph attention network (STGA-Net) for skeleton-based temporal action segmentation. In particular, we propose a spatial-temporal attentive block for prediction generation, which adapts an encoder-decoder architecture, where both the encoder and decoder contain various graph spatial-temporal attention blocks to model the dynamic and non-linear correlation among joints. Experiments on three challenging datasets (PKU-MMD, HuGaDB, and LARa) demonstrate that the performance of our STGA-Net exceeds that of the state-of-the-art and alleviates over-segmentation and ambiguous boundary errors to a large degree.
Prompt What You Need: Enhancing Segmentation in Rainy Scenes with Anchor-Based Prompting

Semantic segmentation in rainy scenes is a challenging task due to the complex environment, class distribution imbalance, and limited annotated data. To address these challenges, we propose a novel framework that utilizes semi-supervised learning and pre-trained segmentation foundation model to achieve superior performance. Specifically, our framework leverages the semi-supervised model as the basis for generating raw semantic segmentation results, while also serving as a guiding force to prompt pre-trained foundation model to compensate for knowledge gaps with entropy-based anchors. In addition, to minimize the impact of irrelevant segmentation masks generated by the pre-trained foundation model, we also propose a mask filtering and fusion mechanism that optimizes raw semantic segmentation results based on the principle of minimum risk. The proposed framework achieves superior segmentation performance on the Rainy WCity dataset and is awarded the first prize in the subtrack of STRAIN in ICME 2023 Grand Challenges.
Normal-Based Global Motion Estimation for LiDAR Point Cloud Lossless Geometry Compression

Point clouds generated by Light Detection And Ranging (LiDAR) are essential for automatic driving. Compression of the LiDAR point cloud faces the challenge of accurate global motion estimation due to the different characteristics of objects in the scene. In this paper, we propose an adaptive global motion estimation algorithm based on normals. First, preprocessing is performed on the point cloud to eliminate points unsuitable for providing global motion. Then, the corresponding point is found according to the different angles of the plane where the object is located. Finally, the global motion estimation is performed according to the correspondence. Experimental results show that the proposed method outperforms the Inter-EM of G-PCC. It can provide an average of 0.24% saving in coding bits on the G-PCC standard dataset and 0.63% saving on the KITTI dataset.
A System for Real-Time Recognition and Fast Retrieval of Close Contacts in Multi-Camera Videos

In recent years, the global community has been grappling with the COVID-19 pandemic. Rapid recognition and retrieval of infected individuals' close contacts could help mitigate the spread of the virus. The task is non-trivial for large scale video due to the tradeoff between effectiveness and efficiency. In this study, we develop a two-stage system that (1) automatically recognizes close contacts in real-time from video footage using multi-object tracking, and (2) fast retrieves the close contacts of a person queried by users in multi-camera videos through an Intra-Camera Association (ICA) approach. We also propose an efficient ICA method based on sparse trajectory correlation graphs. Experimental results demonstrate that our approach exhibits strong performance in terms of both effectiveness and efficiency.
Real Rainy Scene Analysis: A Dual-Module Benchmark for Image Deraining and Segmentation

Scene understanding in adverse weather conditions (e.g., rainy and foggy days) has drawn increasing attention, raising some specific benchmarks and algorithms. However, rain streaks in images and videos can significantly degrade visual quality and reduce the effectiveness of computer vision algorithms. Concerning on lack of rainy-clean pairs training samples in real rainy scenes, we propose an unsupervised deraining method that does not require any explicit rain-clean image pairs for training. Instead, our approach leverages the statistical properties of rain streaks and clean regions to learn a rain removal model. Specifically, we choose contrastive learning that encourages the generator to produce derained images that are indistinguishable from clean images while the discriminator distinguishes between real clean images and rain streak patterns. Furthermore, we incorporate a semi-supervised mechanism during segmentation module that exploits low-confidence Pseudo-label to be treated as negative samples for certain classes to facilitate contrastive learning. Our experiments demonstrate that our integrated framework achieve better performance on the given challenging scenarios such as heavy rain and dynamic scenes.
Comprehensive Augmented Domain Adaptation for Image Segmentation Under Rainy Conditions

This work aims to make full use of a large amount of labeled training data and adapt the knowledge to a small amount of unlabeled data from the perspective of domain adaptation to address the challenges of image segmentation under rainy conditions. We comprehensively examine the existing domain adaptation methods and effectively integrate them, combining their strengths to achieve superior segmentation performance. First, for the backbone, a transformer framework is adopted that takes a transformer encoder and a multilevel context-aware feature fusion decoder. Second, for the learning mechanism, the task entanglement constraint is introduced to prevent the model from overfitting the source domain by separating feature distribution alignment and the segmentation network. Third, a simplex noise transition matrix is employed to model noise distribution and estimate its impact on pseudo-labeled data from the target domain with volume regularization, anchor guidance, and convex guarantee as complementary regularizers. Last, a model aggregation strategy is built empirically that effectively combines the strengths of different domain adaptation models. The combination of these designs leads to a robust and effective final model, demonstrating the superiority of our method in addressing the challenges posed by semantic segmentation in rainy conditions. Our approach demonstrates improved semantic segmentation accuracy in Track I (Semantic Segmentation under Real Rain Scene) and Track III (Semantic Segmentation under Simulated Raw Degradation Scene) of the Challenge STRAIN (Seeing Through the Rain) with the achieved performance of 50.18% and 48.64% in mIoU, respectively.
Predicting Player Performance in Tactical Games with the Lightgbm Algorithm

The accurate prediction of player performance in tactical games is crucial for game designers and developers to improve matchmaking, game balancing, and player engagement. This study presents an approach to predict player performance in Tactical Troops: Anthracite Shift game using the Light Gradient Boosting Machine (LightGBM) algorithm, a scalable and efficient gradient boosting framework. Following data preprocessing, we trained and optimized the LightGBM model using appropriate objective functions and hyperparameter tuning. The proposed approach demonstrated good results in predicting player performance. Our findings highlight the potential of using the LightGBM algorithm in predicting player performance in tactical games, providing valuable insights for game developers to enhance matchmaking, skill assessment, and overall gaming experiences.
sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging

Automatic sleep staging based on electroencephalography (EEG) and electromyography (EMG) signals is an important aspect of sleep-related research. Current sleep staging methods suffer from two major drawbacks. First, there are limited information interactions between modalities in the existing methods. Second, current methods do not develop unified models that can handle different sources of input. To address these issues, we propose a novel sleep stage scoring model sDREAMER, which emphasizes cross-modality interaction and per-channel performance. Specifically, we develop a mixture-of-modality-expert (MoME) model with three pathways for EEG, EMG, and mixed signals with partially shared weights. We further propose a self-distillation training scheme for further information interaction across modalities. Our model is trained with multi-channel inputs and can make classifications on either single-channel or multi-channel inputs. Experiments demonstrate that our model outperforms the existing transformer-based sleep scoring methods for multi-channel inference. For single-channel inference, our model also outperforms the transformer-based models trained with single-channel signals.
Mining Themes in Clinical Notes to Identify Phenotypes and to Predict Length of Stay in Patients admitted with Heart Failure

Heart failure is a syndrome which occurs when the heart is not able to pump blood and oxygen to support other organs in the body. Treatment and management of heart failure in patients include understanding the diagnostic codes and procedure reports of these patients during their hospitalization. Identifying the underlying themes in these diagnostic codes and procedure reports could reveal the clinical phenotypes associated with heart failure. These themes could also help clinicians to predict length of stay in the patients using their clinical notes. Understanding clinical phenotypes on the basis of these themes is important to group patients based on their similar characteristics which could also help in predicting patient outcomes like length of stay. These clinical phenotypes usually have a probabilistic latent structure and hence, as there has been no previous work on identifying phenotypes in clinical notes of heart failure patients using a probabilistic framework and to predict length of stay of these patients using data-driven artificial intelligence-based methods, we apply natural language processing technique, topic modeling, to identify the themes present in diagnostic codes and in procedure reports of 1,200 patients admitted for heart failure at the University of Illinois Hospital and Health Sciences System (UI Health). Topic modeling identified twelve themes each in diagnostic codes and procedure reports. These themes revealed information about different phenotypes related to various perspectives about heart failure, which could help to study patients’ profiles and discover new relationships among medical concepts. Each theme had a set of keywords and each clinical note was labeled with two themes — one corresponding to its diagnostic code and the other corresponding to its procedure reports along with their percentage contribution. We used these themes and their percentage contribution to predict length of stay. We found that the themes discovered in diagnostic codes and procedure reports using topic modeling together were able to predict length of stay of the patients with an accuracy of 61.1% and an Area under the Receiver Operating Characteristic Curve (ROC AUC) value of 0.828.
A Generic Framework to Develop and Verify Security Mechanisms at the Microarchitectural Level: Application to Control-Flow Integrity

In recent years, the disclosure of several significant security vulnerabilities has revealed the trust put in some presumed security properties of commonplace hardware to be misplaced. We propose to design hardware systems with security mechanisms, together with a formal statement of the security properties obtained, and a machine-checked proof that the hardware security mechanisms indeed implement the sought-for security property. Formally proving security properties about hardware systems might seem prohibitively complex and expensive. In this paper, we tackle this concern by designing a realistic and accessible methodology on top of the Kôlka Hardware Description Language for specifying and proving security properties during hardware development. Our methodology is centered around a verified compiler from high-level and inefficient to work with Kôlka models to an equivalent lower-level representation, where side effects are made explicit and reasoning is convenient. We apply this methodology to a concrete example: the formal specification and implementation of a shadow stack mechanism on an RV32I processor. We prove that this security mechanism is correct, i.e., any illegal modification of a return address does indeed result in the termination of the whole system. Furthermore, we show that this modification of the processor does not impact its behaviour in other, unexpected ways.
SAGE-NDVI: A Stereotype-Breaking Evaluation Metric for Remote Sensing Image Dehazing Using Satellite-to-Ground NDVI Knowledge

Image dehazing is a meaningful low-level computer vision task and can be applied to a variety of contexts. In our industrial deployment scenario based on remote sensing (RS) images, the quality of image dehazing directly affects the grade of our crop identification and growth monitoring products. However, the widely used peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) provide ambiguous visual interpretation. In this paper, we design a new objective metric for RS image dehazing evaluation. Our proposed metric leverages a ground-based phenology observation resource to calculate the vegetation index error between RS and ground images at a hazy date. Extensive experiments validate that our metric appropriately evaluates different dehazing models and is in line with human visual perception.
Fostering Human Activity Recognition Workflows: An Open-Source Baseline Framework

The application of machine and deep learning algorithms in Human Activity Recognition (HAR) has shown great potential for monitoring various professional and daily life activities, benefiting different research areas such as healthcare, well-being and industrial automation. HAR can enable the development of various services and applications to empower technical performance and enable risk prevention in working places, to support education and training, and, more in general, to monitor the biopsychosocial status of people. However, we still lack a baseline framework for easily implementing the data processing pipeline that must be designed to setup and configure HAR workflows. This makes challenging to estimate the effectiveness, efficiency, and the overall quality of HAR solutions, thus hindering the comparison among different approaches. This also increases the likelihood that researchers introduce errors, which negatively affect the accuracy of the obtained results. To fill in the gap, this paper introduces B-HAR, an open-source framework to automatically implement baseline HAR workflows.
Old-Photo Restoration with Detail- and Structure-Enhanced Cascaded Learning

Restoring old photos with unknown and complex degradations remains proverbially meaningful but challenging today. The related image inpainting and image restoration methods can not address structured and unstructured defects in old photos simultaneously, which will produce over-smoothed, lacking well-preserved structure and outdated results. In this paper, we propose a cascaded two-stage old photo restoration method: the coarse stage mainly restores unstructured defects by learning the mapping function between the degraded image domain and the clean image domain; the refinement stage focuses on filling in missing or damaged regions with a multi-scale residual dense edge restoration network and a Fourier inpainting network. Particularly, we embed simple yet effective feature extraction modules to retain detail features in the coarse stage and extract the structure information from the edge map in the refinement stage. Experiments show that, with dedicated designs, our approach surpasses the-state-of-art old photo restoration methods qualitatively and quantitatively.
SBSR: A Simple Residual Network for Efficient Burst Super-Resolution

Burst Super-Resolution (BurstSR) attempts to restore a high resolution image from a misaligned, low-resolution RAW burst sequence. Current models for BurstSR are complex and require significant computational resources for training and practical application. This paper introduces a new, simplified model called Simple Residual Network for Burst Super-Resolution (SBSR), which includes multiple Simple Residual Networks (SRNs) that not only greatly reduce the parameters due to their simple structure, but also speed up the training and inference process with excellent feature integration ability, while maintaining good performance. Furthermore, we propose a Pre-trained Shallow Feature Extractor (PSFE) for contrastive loss to further improve the performance of SBSR. Experimental results show that SBSR surpasses existing methods in terms of the trade-off between performance and efficiency. Code is available at https://github.com/githublei-min/SBSR.
Decomposed Key-Point Detector for Swimming Pool Localization

Pool localization is an essential prerequisite for swimmer analysis or performance measurement. Automated analysis studies on swimming pools have often been proposed due to the increase in broadcast videos and the development of machine learning techniques, but there have been few studies dealing with GPU memory usage and latency in terms of practical analysis. This work proposes an efficient swimming pool key-point detection method utilizing a U-Net-based decomposed detector, for robust detection regardless of camera parameters. Experiments show the proposed detector has higher accuracy than the original model, while reducing latency and memory usage by 4.80 and 2.91 times, respectively.