Reviewerly/exHarmony huggingface dataset:
Nighttime Visibility Enhancement by Increasing the Dynamic Range and Suppression of Light Effects

Most existing nighttime visibility enhancement methods focus on low light. Night images, however, do not only suffer from low light, but also from man-made light effects such as glow, glare, floodlight, etc. Hence, when the existing nighttime visibility enhancement methods are applied to these images, they intensify the effects, degrading the visibility even further. High dynamic range (HDR) imaging methods can address the low light and over-exposed regions, however they cannot remove the light effects, and thus cannot enhance the visibility in the affected regions. In this paper, given a single nighttime image as input, our goal is to enhance its visibility by increasing the dynamic range of the intensity, and thus can boost the intensity of the low light regions, and at the same time, suppress the light effects (glow, glare) simultaneously. First, we use a network to estimate the camera response function (CRF) from the input image to linearise the image. Second, we decompose the linearised image into low-frequency (LF) and high-frequency (HF) feature maps that are processed separately through two networks for light effects suppression and noise removal respectively. Third, we use a network to increase the dynamic range of the processed LF feature maps, which are then combined with the processed HF feature maps to generate the final output that has increased dynamic range and suppressed light effects. Our experiments show the effectiveness of our method in comparison with the state-of-the-art nighttime visibility enhancement methods.
VSCL: Automating Vulnerability Detection in Smart Contracts with Deep Learning

With the increase of the adoption of blockchain technology in providing decentralized solutions to various problems, smart contracts have become more popular to the point that billions of US Dollars are currently exchanged every day through such technology. Meanwhile, various vulnerabilities in smart contracts have been exploited by attackers to steal cryptocurrencies worth millions of dollars. The automatic detection of smart contract vulnerabilities therefore is an essential research problem. Existing solutions to this problem particularly rely on human experts to define features or different rules to detect vulnerabilities. However, this often causes many vulnerabilities to be ignored, and they are inefficient in detecting new vulnerabilities. In this study, to overcome such challenges, we propose the VSCL framework to automatically detect vulnerabilities in smart contracts on the blockchain. More specifically, first, we utilize novel feature vector generation techniques from bytecode of smart contract since the source code of smart contracts are rarely available in public. Next, the collected vectors are fed into our novel metric learning-based deep neural network(DNN) to get the detection result. We conduct comprehensive experiments on a large-scale benchmark, and the quantitative results demonstrate the effectiveness and efficiency of our approach.
Economic analysis of hydrogen-powered data center

The data center needs more and more electricity due to the explosive growth of IT servers and it could cause electricity power shortage and huge carbon emission. It is an attractive and promising solution to power the data center with hydrogen energy source. The present work aims to conduct an economic analysis on the hydrogen-powered data center. Configurations of hydrogen-powered and traditional data centers are compared and the differences focus on backup power system, converter/inverter, fuel cell subsystem, carbon emission, hydrogen and electricity consumptions. Economic analysis is conducted to evaluate the feasibility to power the data center with hydrogen energy source. Results show that electricity price increasing rate and hydrogen cost are the main factors to influence economic feasibility of hydrogen-powered data center. When the electricity price keeps constant in the coming two decades, the critical hydrogen price is about 2.8 U.S. dollar per kilogram. If the electricity price could increase 5% annually due to explosive growth of electric vehicles and economy, critical hydrogen price will become 6.4 U.S. dollar per kilogram. Hydrogen sources and transportation determine the hydrogen price together. Hydrogen production cost varies greatly with hydrogen sources and production technologies. Hydrogen transport cost is greatly influenced by distances and H2 consumptions to consumers. It could be summarized that the hydrogen-powered data center is economic if hydrogen could be produced from natural gas or H2-rich industrial waste streams in chemical plant and data center could not be built too far away from hydrogen sources. In addition, large-scale hydrogen-powered data center is more likely to be economic. Solar hydrogen powered data center has entered into a critical stage in the economic feasibility. Solar hydrogen production cost has restrained the H2 utilization in data center power systems now, since it could be competitive only when more strict carbon emission regulation is employed, hydrogen production cost reduces greatly and electricity price is increasing greatly in the future. However, it could be expected solar hydrogen-powered system will be adopted as the power source of data centers in the next few years.
Multimodal Learning For Hateful Memes Detection

Memes are used for spreading ideas through social networks. Although most memes are created for humor, some memes become hateful under the combination of pictures and text. Automatically detecting hateful memes can help reduce their harmful social impact. Compared to the conventional multimodal tasks, where the visual and textual information is semantically aligned, hateful memes detection is a more challenging task since the image and text in memes are weakly aligned or even irrelevant. Thus it requires the model to have a deep understanding of the content and perform reasoning over multiple modalities. This paper focuses on multimodal hateful memes detection and proposes a novel method incorporating the image captioning process into the memes detection process. We conduct extensive experiments on multimodal meme datasets and illustrate the effectiveness of our approach. Our model achieves promising results on the Hateful Memes Detection Challenge. Our code is made publicly available at GitHub.
Post-hoc Uncertainty Calibration for Domain Drift Scenarios

We address the problem of uncertainty calibration. While standard deep neural networks typically yield uncalibrated predictions, calibrated confidence scores that are representative of the true likelihood of a prediction can be achieved using post-hoc calibration methods. However, to date, the focus of these approaches has been on in-domain calibration. Our contribution is two-fold. First, we show that existing post-hoc calibration methods yield highly over-confident predictions under domain shift. Second, we introduce a simple strategy where perturbations are applied to samples in the validation set before performing the post-hoc calibration step. In extensive experiments, we demonstrate that this perturbation step results in substantially better calibration under domain shift on a wide range of architectures and modelling tasks.
Quality control of microseismic P-phase arrival picks in coal mine based on machine learning

Microseismic events generally contain strong noise-polluting and unobvious P-phase oscillating channel waveforms. The automatic P-phase arrival picking accuracy of these channel waveforms tends to be low, or even are false. Currently, unusable P-picks are not screened out automatically before geophysics inversions in most microseismic data processing software. Therefore, manual interventions are needed to remove or correct the unusable P-picks. However, rapidly increasing monitoring data causes manual handling to be time-consuming and lagging. Supervised machine learning (ML) is applied to distinguish useable and unusable P-picks automatically. Big data analysis revealed that the waveform features, including signal-to-noise ratio, signal-to-noise variance ratio, P-wave starting-up slope, and peak amplitude have impact on P-pick accuracy. In contrast, the effect of the short-time zero-crossing rate on the P-pick accuracy is not as obvious. Five P-pick quality control models were trained based on traditional machine learning approaches, including discriminant analysis, logistic regression, k-nearest neighbor, support vector machine, and Naive Bayes classifier. For these five models, the input data are P-pick labels and waveform features. In addition, another P-pick quality control model was trained based on convolutional neural network. While, the input data are P-pick images and labels. The training sets used in all six machine learning models are uniform. The testing experiments with uniform testing set show that the support vector machine generated best the performance among traditional machine learning approaches, with 82.81% accuracy. However, the convolutional neural network model generated outstanding performance in recognizing P-pick, with 91.71% accuracy. The automatic P-pick quality control method proposed in this study can facilitate the precision and efficiency of the automatic processing of microseismic signals.
Revisiting Huffman Coding: Toward Extreme Performance on Modern GPU Architectures

Today's high-performance computing (HPC) applications are producing vast volumes of data, which are challenging to store and transfer efficiently during the execution, such that data compression is becoming a critical technique to mitigate the storage burden and data movement cost. Huffman coding is arguably the most efficient Entropy coding algorithm in information theory, such that it could be found as a fundamental step in many modern compression algorithms such as DEFLATE. On the other hand, today's HPC applications are more and more relying on the accelerators such as GPU on supercomputers, while Huffman encoding suffers from low throughput on GPUs, resulting in a significant bottleneck in the entire data processing. In this paper, we propose and implement an efficient Huffman encoding approach based on modern GPU architectures, which addresses two key challenges: (1) how to parallelize the entire Huffman encoding algorithm, including codebook construction, and (2) how to fully utilize the high memory-bandwidth feature of modern GPU architectures. The detailed contribution is fourfold. (1) We develop an efficient parallel codebook construction on GPUs that scales effectively with the number of input symbols. (2) We propose a novel reduction based encoding scheme that can efficiently merge the codewords on GPUs. (3) We optimize the overall GPU performance by leveraging the state-of-the-art CUDA APIs such as Cooperative Groups. (4) We evaluate our Huffman encoder thoroughly using six real-world application datasets on two advanced GPUs and compare with our implemented multithreaded Huffman encoder. Experiments show that our solution can improve the encoding throughput by up to 5.0× and 6.8× on NVIDIA RTX 5000 and V100, respectively, over the state-of-the-art GPU Huffman encoder, and by up to 3.3× over the multithread encoder on two 28-core Xeon Platinum 8280 CPUs.
Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry

We present a new generic method for shadow-aware multi-view satellite photogrammetry of Earth Observation scenes. Our proposed method, the Shadow Neural Radiance Field (S-NeRF) follows recent advances in implicit volumetric representation learning. For each scene, we train S-NeRF using very high spatial resolution optical images taken from known viewing angles. The learning requires no labels or shape priors: it is self-supervised by an image reconstruction loss. To accommodate for changing light source conditions both from a directional light source (the Sun) and a diffuse light source (the sky), we extend the NeRF approach in two ways. First, direct illumination from the Sun is modeled via a local light source visibility field. Second, indirect illumination from a diffuse light source is learned as a non-local color field as a function of the position of the Sun. Quantitatively, the combination of these factors reduces the altitude and color errors in shaded areas, compared to NeRF. The S-NeRF methodology not only performs novel view synthesis and full 3D shape estimation, it also enables shadow detection, albedo synthesis, and transient object filtering, without any explicit shape supervision.
Learning Scale-Consistent Attention Part Network for Fine-Grained Image Recognition

Discriminative region localization and feature learning are crucial for fine-grained visual recognition. Existing approaches solve this issue by attention mechanism or part based methods while neglecting consistency between attention and local parts, as well as the rich relation information among parts. This paper proposes a Scale-consistent Attention Part Network (SCAPNet) to address that issue, which seamlessly integrates three novel modules: grid gate attention unit (gGAU), scale-consistent attention part selection (SCAPS), and part relation modeling (PRM). The gGAU module represents the grid region at a certain fine-scale with middle layer CNN features and produces hard attention maps with the lightweight Gumbel-Max based gate. The SCAPS module utilizes attention to guide part selection across multi-scales and keep the selection scale-consistent. The PRM module utilizes the self-attention mechanism to build the relationship among parts based on their appearance and relative geo-positions. SCAPNet can be learned in an end-to-end way and demonstrates state-of-the-art accuracy on several publicly available fine-grained recognition datasets (CUB-200-2011, FGVC-Aircraft, Veg200, and Fru92).
TGRNet: A Table Graph Reconstruction Network for Table Structure Recognition

A table arranging data in rows and columns is a very effective data structure, which has been widely used in business and scientific research. Considering large-scale tabular data in online and offline documents, automatic table recognition has attracted increasing attention from the document analysis community. Though human can easily understand the structure of tables, it remains a challenge for machines to understand that, especially due to a variety of different table layouts and styles. Existing methods usually model a table as either the markup sequence or the adjacency matrix between different table cells, failing to address the importance of the logical location of table cells, e.g., a cell is located in the first row and the second column of the table. In this paper, we reformulate the problem of table structure recognition as the table graph reconstruction, and propose an end-to-end trainable table graph reconstruction network (TGRNet) for table structure recognition. Specifically, the proposed method has two main branches, a cell detection branch and a cell logical location branch, to jointly predict the spatial location and the logical location of different cells. Experimental results on three popular table recognition datasets and a new dataset with table graph annotations (TableGraph-350K) demonstrate the effectiveness of the proposed TGRNet for table structure recognition. Code and annotations will be made publicly available at https://github.com/xuewenyuan/TGRNet.
Repetitive Activity Counting by Sight and Sound

This paper strives for repetitive activity counting in videos. Different from existing works, which all analyze the visual video content only, we incorporate for the first time the corresponding sound into the repetition counting process. This benefits accuracy in challenging vision conditions such as occlusion, dramatic camera view changes, low resolution, etc. We propose a model that starts with analyzing the sight and sound streams separately. Then an audiovisual temporal stride decision module and a reliability estimation module are introduced to exploit cross-modal temporal interaction. For learning and evaluation, an existing dataset is repurposed and reorganized to allow for repetition counting with sight and sound. We also introduce a variant of this dataset for repetition counting under challenging vision conditions. Experiments demonstrate the benefit of sound, as well as the other introduced modules, for repetition counting. Our sight-only model already outperforms the state-of-the-art by itself, when we add sound, results improve notably, especially under harsh vision conditions. The code and datasets are available at https://github.com/xiaobai1217/RepetitionCounting.
MPdist-based missing data imputation for supporting big data analyses in IoT-based applications

Recent years have witnessed an enormous growth in the number of wireless IoT devices and thereby Internet of Things (IoT) is classified as one of the novel cutting edge technologies that has redesigned the traditional industry into a smart industry, given its applicability for a wide range of applications in providing data-driven decision-making. Meanwhile, the data with missing values is still emerging as one of the longstanding challenges in an IoT architecture due to the common-mode failures, potentially leading to both bias and loss of precision. In spite of the fact that numerous techniques have been developed for imputing missing values, the major issue in terms of imputation precision or computational complexity for large missing subsequences is still a matter under debate. To address this issue, this paper proposes a newly developed algorithm called MP-BMDI that ensures high imputation performance for supporting big data analyses in IoT-based applications, where the absence of large missing subsequences is fully required to offer unbiased results. In our approach, we initially seek a finite number of subsequences that are mostly similar to the subsequence before the missing values, then adjust the height of these following subsequences to optimal locations. Once the most proper subsequence for replacing is chosen among them based on the pattern score function PSF(r) introduced in this paper, the missing gap is completely filled by the corresponding subsequence. Numerical results are here presented to validate the merits of the proposed algorithm compared to the alternative benchmark approaches by leveraging sensor data collected from real-time environmental monitoring and deliver significant insights on the effectiveness of the proposed algorithm from various perspectives.
Exploring the Effects of Natural Language Justifications in Food Recommender Systems

Users of food recommender systems typically prefer popular recipes, which tend to be unhealthy. To encourage users to select healthier recommendations by making more informed food decisions, we introduce a methodology to generate and present a natural language justification that emphasizes the nutritional content, or health risks and benefits of recommended recipes. We designed a framework that takes a user and two food recommendations as input and produces an automatically generated natural language justification as output, which is based on the user's characteristics and the recipes' features. In doing so, we implemented and evaluated eight different justification strategies through two different justification styles (e.g., comparing each recipe's food features) in an online user study (N = 503). We compared user food choices for two personalized recommendation approaches, popularity-based vs our health-aware algorithm, and evaluated the impact of presenting natural language justifications. We showed that comparative justifications styles are effective in supporting choices for our healthy-aware recommendations, confirming the impact of our methodology on food choices.
Recognition of user-dependent and independent static hand gestures: Application to sign language

Static hand gesture (HG) recognition for both user-dependent and user-independent is a challenging problem, especially when there are changes in lighting, hand position, and background, the recognition becomes more complex. To solve this problem, this paper proposes a static hand gesture recognition based on a set of image descriptors: Gradient Local Auto-Correlation (GLAC), Gabor Wavelet Transform (GWT), and Fast Discrete Curve Transform (FDCT). Principal Component Analysis (PCA) was used to reduce dimensionality. Tests were performed on three sign language datasets and one hand posture dataset using neural network classifiers, K-Nearest Neighbor (KNN) classifiers, and combined classifiers. The results obtained were compared to the state of the art and show an accuracy of 100% for user-independent and 98.33% for user-dependent gestures, despite the difficult acquisition conditions of the datasets.
FAST: FPGA-based Subgraph Matching on Massive Graphs

Subgraph matching is a basic operation widely used in many applications. However, due to its NP-hardness and the explosive growth of graph data, it is challenging to compute subgraph matching, especially in large graphs. In this paper, we aim at scaling up subgraph matching on a single machine using FPGAs. Specifically, we propose a CPU-FPGA co-designed framework. On the CPU side, we first develop a novel auxiliary data structure called candidate search tree (CST) which serves as a complete search space of subgraph matching. CST can be partitioned and fully loaded into FPGAs' on-chip memory. Then, a workload estimation technique is proposed to balance the load between the CPU and FPGA. On the FPGA side, we design and implement the first FPGA-based subgraph matching algorithm, called FAST. To take full advantage of the pipeline mechanism on FPGAs, task parallelism optimization and task generator separation strategy are proposed for FAST, achieving massive parallelism. Moreover, we carefully develop a BRAM-only matching process to fully utilize FPGA's on-chip memory, which avoids the expensive intermediate data transfer between FPGA's BRAM and DRAM. Comprehensive experiments show that FAST achieves up to 462.0x and 150.0x speedup compared with the state-of-the-art algorithm DAF and CECI, respectively. In addition, FAST is the only algorithm that can handle the billion-scale graph using one machine in our experiments.
Multi-Scale Hourglass Hierarchical Fusion Network for Single Image Deraining

Rain streaks bring serious blurring and visual quality degradation, which often vary in size, direction and density. Current CNN-based methods achieve encouraging performance, while are limited to depict rain characteristics and recover image details in the poor visibility environment. To address these issues, we present a Multi-scale Hourglass Hierarchical Fusion Network (MH <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> F-Net) in end-to-end manner, to exactly captures rain streak features with multi-scale extraction, hierarchical distillation and information aggregation. For better extracting the features, a novel Multi-scale Hourglass Extraction Block (MHEB) is proposed to get local and global features across different scales through down- and up-sample process. Besides, a Hierarchical Attentive Distillation Block (HADB) then employs the dual attention feature responses to adaptively recalibrate the hierarchical features and eliminate the redundant ones. Further, we introduce a Residual Projected Feature Fusion (RPFF) strategy to progressively discriminate feature learning and aggregate different features instead of directly concatenating or adding. Extensive experiments on both synthetic and real rainy datasets demonstrate the effectiveness of the designed MH <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> F-Net by comparing with recent state-of-the-art deraining algorithms. Our source code will be available on the GitHub: https://github.com/cxtalk/MH2F-Net.