OpenRaiser/Intern-Atlas huggingface dataset:
FakeSV: A Multimodal Benchmark with Rich Social Context for Fake News Detection on Short Video Platforms
Venue: arXiv cs.MM 2022
Year: 2022

Short video platforms have become an important channel for news sharing, but also a new breeding ground for fake news. To mitigate this problem, research of fake news video detection has recently received a lot of attention. Existing works face two roadblocks: the scarcity of comprehensive and largescale datasets and insufficient utilization of multimodal information. Therefore, in this paper, we construct the largest Chinese short video dataset about fake news named FakeSV, which includes news content, user comments, and publisher profiles simultaneously. To understand the characteristics of fake news videos, we conduct exploratory analysis of FakeSV from different perspectives. Moreover, we provide a new multimodal detection model named SV-FEND, which exploits the cross-modal correlations to select the most informative features and utilizes the social context information for detection. Extensive experiments evaluate the superiority of the proposed method and provide detailed comparisons of different methods and modalities for future works.
Investigating the Cybersecurity of Smart Grids Based on Cyber-Physical Twin Approach
Venue: arXiv cs.CR 2022
Year: 2022

While the increasing penetration of information and communication technology into distribution grid brings numerous benefits, it also opens up a new threat landscape, particularly through cyberattacks. To provide a basis for countermeasures against such threats, this paper addresses the investigation of the impact and manifestations of cyberattacks on smart grids by replicating the power grid in a secure, isolated, and controlled laboratory environment as a cyber-physical twin. Currently, detecting intrusions by unauthorized third parties into the central monitoring and control system of grid operators, especially attacks within the grid perimeter, is a major challenge. The development and validation of methods to detect and prevent coordinated and timed attacks on electric power systems depends not only on the availability and quality of data from such attack scenarios, but also on suitable realistic investigation environments. However, to create a comprehensive investigation environment, a realistic representation of the study object is required to thoroughly investigate critical cyberattacks on grid operations and evaluate their impact on the power grid using real data. In this paper, we demonstrate our cyber-physical twin approach using a microgrid in the context of a cyberattack case study.
Modeling Fine-grained Information via Knowledge-aware Hierarchical Graph for Zero-shot Entity Retrieval
Venue: arXiv cs.CL 2022
Year: 2022

Zero-shot entity retrieval, aiming to link mentions to candidate entities under the zero-shot setting, is vital for many tasks in Natural Language Processing. Most existing methods represent mentions/entities via the sentence embeddings of corresponding context from the Pre-trained Language Model. However, we argue that such coarse-grained sentence embeddings can not fully model the mentions/entities, especially when the attention scores towards mentions/entities are relatively low. In this work, we propose GER, a \textbf{G}raph enhanced \textbf{E}ntity \textbf{R}etrieval framework, to capture more fine-grained information as complementary to sentence embeddings. We extract the knowledge units from the corresponding context and then construct a mention/entity centralized graph. Hence, we can learn the fine-grained information about mention/entity by aggregating information from these knowledge units. To avoid the graph information bottleneck for the central mention/entity node, we construct a hierarchical graph and design a novel Hierarchical Graph Attention Network~(HGAN). Experimental results on popular benchmarks demonstrate that our proposed GER framework performs better than previous state-of-the-art models. The code has been available at https://github.com/wutaiqiang/GER-WSDM2023.
Mission-Aware Value of Information Censoring for Distributed Filtering
Venue: arXiv cs.RO 2022
Year: 2022

In this paper, we study the problem of distributed estimation with an emphasis on communication-efficiency. The proposed algorithm is based on a windowed maximum a posteriori (MAP) estimation problem, wherein each agent in the network locally computes a Kalman-like filter estimate that approximates the centralized MAP solution. Information sharing among agents is restricted to their neighbors only, with guarantees on overall estimate consistency provided via logarithmic opinion pooling. The problem is efficiently distributed using the alternating direction method of multipliers (ADMM), whose overall communication usage is further reduced by a value of information (VoI) censoring mechanism, wherein agents only transmit their primal-dual iterates when deemed valuable to do so. The proposed censoring mechanism is mission-aware, enabling a globally efficient use of communication resources while guaranteeing possibly different local estimation requirements. To illustrate the validity of the approach we perform simulations in a target tracking scenario.
Patch-level Gaze Distribution Prediction for Gaze Following
Venue: arXiv cs.CV 2022
Year: 2022

Gaze following aims to predict where a person is looking in a scene, by predicting the target location, or indicating that the target is located outside the image. Recent works detect the gaze target by training a heatmap regression task with a pixel-wise mean-square error (MSE) loss, while formulating the in/out prediction task as a binary classification task. This training formulation puts a strict, pixel-level constraint in higher resolution on the single annotation available in training, and does not consider annotation variance and the correlation between the two subtasks. To address these issues, we introduce the patch distribution prediction (PDP) method. We replace the in/out prediction branch in previous models with the PDP branch, by predicting a patch-level gaze distribution that also considers the outside cases. Experiments show that our model regularizes the MSE loss by predicting better heatmap distributions on images with larger annotation variances, meanwhile bridging the gap between the target prediction and in/out prediction subtasks, showing a significant improvement in performance on both subtasks on public gaze following datasets.
A review of laser scanning for geological and geotechnical applications in underground mining
Venue: arXiv cs.CV 2022
Year: 2022

Laser scanning can provide timely assessments of mine sites despite adverse challenges in the operational environment. Although there are several published articles on laser scanning, there is a need to review them in the context of underground mining applications. To this end, a holistic review of laser scanning is presented including progress in 3D scanning systems, data capture/processing techniques and primary applications in underground mines. Laser scanning technology has advanced significantly in terms of mobility and mapping, but there are constraints in coherent and consistent data collection at certain mines due to feature deficiency, dynamics, and environmental influences such as dust and water. Studies suggest that laser scanning has matured over the years for change detection, clearance measurements and structure mapping applications. However, there is scope for improvements in lithology identification, surface parameter measurements, logistic tracking and autonomous navigation. Laser scanning has the potential to provide real-time solutions but the lack of infrastructure in underground mines for data transfer, geodetic networking and processing capacity remain limiting factors. Nevertheless, laser scanners are becoming an integral part of mine automation thanks to their affordability, accuracy and mobility, which should support their widespread usage in years to come.
LHDR: HDR Reconstruction for Legacy Content using a Lightweight DNN
Venue: arXiv cs.CV 2022
Year: 2022

High dynamic range (HDR) image is widely-used in graphics and photography due to the rich information it contains. Recently the community has started using deep neural network (DNN) to reconstruct standard dynamic range (SDR) images into HDR. Albeit the superiority of current DNN-based methods, their application scenario is still limited: (1) heavy model impedes real-time processing, and (2) inapplicable to legacy SDR content with more degradation types. Therefore, we propose a lightweight DNN-based method trained to tackle legacy SDR. For better design, we reform the problem modeling and emphasize degradation model. Experiments show that our method reached appealing performance with minimal computational cost compared with others.
Revealing intra-urban spatial structure through an exploratory analysis by combining road network abstraction model and taxi trajectory data
Venue: arXiv cs.SI 2022
Year: 2022

The unprecedented urbanization in China has dramatically changed the urban spatial structure of cities. With the proliferation of individual-level geospatial big data, previous studies have widely used the network abstraction model to reveal the underlying urban spatial structure. However, the construction of network abstraction models primarily focuses on the topology of the road network without considering individual travel flows along with the road networks. Individual travel flows reflect the urban dynamics, which can further help understand the underlying spatial structure. This study therefore aims to reveal the intra-urban spatial structure by integrating the road network abstraction model and individual travel flows. To achieve this goal, we 1) quantify the spatial interaction relatedness of road segments based on the Word2Vec model using large volumes of taxi trip data, then 2) characterize the road abstraction network model according to the identified spatial interaction relatedness, and 3) implement a community detection algorithm to reveal sub-regions of a city. Our results reveal three levels of hierarchical spatial structures in the Wuhan metropolitan area. This study provides a data-driven approach to the investigation of urban spatial structure via identifying traffic interaction patterns on the road network, offering insights to urban planning practice and transportation management.
H-VFI: Hierarchical Frame Interpolation for Videos with Large Motions
Venue: arXiv cs.CV 2022
Year: 2022

Capitalizing on the rapid development of neural networks, recent video frame interpolation (VFI) methods have achieved notable improvements. However, they still fall short for real-world videos containing large motions. Complex deformation and/or occlusion caused by large motions make it an extremely difficult problem in video frame interpolation. In this paper, we propose a simple yet effective solution, H-VFI, to deal with large motions in video frame interpolation. H-VFI contributes a hierarchical video interpolation transformer (HVIT) to learn a deformable kernel in a coarse-to-fine strategy in multiple scales. The learnt deformable kernel is then utilized in convolving the input frames for predicting the interpolated frame. Starting from the smallest scale, H-VFI updates the deformable kernel by a residual in succession based on former predicted kernels, intermediate interpolated results and hierarchical features from transformer. Bias and masks to refine the final outputs are then predicted by a transformer block based on interpolated results. The advantage of such a progressive approximation is that the large motion frame interpolation problem can be decomposed into several relatively simpler sub-tasks, which enables a very accurate prediction in the final results. Another noteworthy contribution of our paper consists of a large-scale high-quality dataset, YouTube200K, which contains videos depicting a great variety of scenarios captured at high resolution and high frame rate. Extensive experiments on multiple frame interpolation benchmarks validate that H-VFI outperforms existing state-of-the-art methods especially for videos with large motions.
BrainTTA: A 35 fJ/op Compiler Programmable Mixed-Precision Transport-Triggered NN SoC
Venue: arXiv cs.AR 2022
Year: 2022

Recently, accelerators for extremely quantized deep neural network (DNN) inference with operand widths as low as 1-bit have gained popularity due to their ability to largely cut down energy cost per inference. In this paper, a flexible SoC with mixed-precision support is presented. Contrary to the current trend of fixed-datapath accelerators, this architecture makes use of a flexible datapath based on a Transport-Triggered Architecture (TTA). The architecture is fully programmable using C. The accelerator has a peak energy efficiency of 35/67/405 fJ/op (binary, ternary, and 8-bit precision) and a throughput of 614/307/77 GOPS, which is unprecedented for a programmable architecture.
Instance-specific and Model-adaptive Supervision for Semi-supervised Semantic Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

Recently, semi-supervised semantic segmentation has achieved promising performance with a small fraction of labeled data. However, most existing studies treat all unlabeled data equally and barely consider the differences and training difficulties among unlabeled instances. Differentiating unlabeled instances can promote instance-specific supervision to adapt to the model's evolution dynamically. In this paper, we emphasize the cruciality of instance differences and propose an instance-specific and model-adaptive supervision for semi-supervised semantic segmentation, named iMAS. Relying on the model's performance, iMAS employs a class-weighted symmetric intersection-over-union to evaluate quantitative hardness of each unlabeled instance and supervises the training on unlabeled data in a model-adaptive manner. Specifically, iMAS learns from unlabeled instances progressively by weighing their corresponding consistency losses based on the evaluated hardness. Besides, iMAS dynamically adjusts the augmentation for each instance such that the distortion degree of augmented instances is adapted to the model's generalization capability across the training course. Not integrating additional losses and training procedures, iMAS can obtain remarkable performance gains against current state-of-the-art approaches on segmentation benchmarks under different semi-supervised partition protocols.
Estimating the Effective Support Size in Constant Query Complexity
Venue: arXiv cs.DS 2022
Year: 2022

Estimating the support size of a distribution is a well-studied problem in statistics. Motivated by the fact that this problem is highly non-robust (as small perturbations in the distributions can drastically affect the support size) and thus hard to estimate, Goldreich [ECCC 2019] studied the query complexity of estimating the $ε$-\emph{effective support size} $\text{Ess}_ε$ of a distribution ${P}$, which is equal to the smallest support size of a distribution that is $ε$-far in total variation distance from ${P}$. In his paper, he shows an algorithm in the dual access setting (where we may both receive random samples and query the sampling probability $p(x)$ for any $x$) for a bicriteria approximation, giving an answer in $[\text{Ess}_{(1+β)ε},(1+γ) \text{Ess}_ε]$ for some values $β, γ> 0$. However, his algorithm has either super-constant query complexity in the support size or super-constant approximation ratio $1+γ= ω(1)$. He then asked if this is necessary, or if it is possible to get a constant-factor approximation in a number of queries independent of the support size. We answer his question by showing that not only is complexity independent of $n$ possible for $γ>0$, but also for $γ=0$, that is, that the bicriteria relaxation is not necessary. Specifically, we show an algorithm with query complexity $O(\frac{1}{β^3 ε^3})$. That is, for any $0 < ε, β< 1$, we output in this complexity a number $\tilde{n} \in [\text{Ess}_{(1+β)ε},\text{Ess}_ε]$. We also show that it is possible to solve the approximate version with approximation ratio $1+γ$ in complexity $O\left(\frac{1}{β^2 ε} + \frac{1}{βεγ^2}\right)$. Our algorithm is very simple, and has $4$ short lines of pseudocode.
Rooms with Text: A Dataset for Overlaying Text Detection
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we introduce a new dataset of room interior pictures with overlaying and scene text, totalling to 4836 annotated images in 25 product categories. We provide details on the collection and annotation process of our dataset, and analyze its statistics. Furthermore, we propose a baseline method for overlaying text detection, that leverages the character region-aware text detection framework to guide the classification model. We validate our approach and show its efficiency in terms of binary classification metrics, reaching the final performance of 0.95 F1 score, with false positive and false negative rates of 0.02 and 0.06 correspondingly.
A Tale of Frozen Clouds: Quantifying the Impact of Algorithmic Complexity Vulnerabilities in Popular Web Servers
Venue: arXiv cs.CR 2022
Year: 2022

Algorithmic complexity vulnerabilities are a class of security problems that enables attackers to trigger the worst-case complexity of certain algorithms. Such vulnerabilities can be leveraged to deploy low-volume, asymmetric, CPU-based denial-of-service (DoS) attacks. Previous work speculates that these vulnerabilities are more dangerous in certain web servers, like Node.js, than in traditional ones, like Apache. We believe it is of utmost importance to understand if this is indeed the case or if there are ways to compensate against such problems using various deployment strategies. To this end, we study the resilience of popular web servers against CPU-based DoS attacks in four major cloud platforms under realistic deployment conditions. We find that there are indeed significant differences in how various web servers react to an attack. However, our results suggest a more nuanced landscape than previously believed: while event-based systems tend to recover faster from DoS in certain scenarios, they also suffer the worst performance degradation overall. Nevertheless, in some setups, Apache performs worse than event-based systems, and there are cloud platforms in which all the considered servers are seriously exposed to the attack. We also find that developers can harden their servers against CPU-based DoS attacks by increasing the number of server instances running in parallel. This, in turn, can lead to an increased cost of operation or a slight degradation of performance in non-DoS conditions.
LISA: Localized Image Stylization with Audio via Implicit Neural Representation
Venue: arXiv cs.CV 2022
Year: 2022

We present a novel framework, Localized Image Stylization with Audio (LISA) which performs audio-driven localized image stylization. Sound often provides information about the specific context of the scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. In this work, we propose a framework that a user provides an audio input to localize the sound source in the input image and another for locally stylizing the target object or scene. LISA first produces a delicate localization map with an audio-visual localization network by leveraging CLIP embedding space. We then utilize implicit neural representation (INR) along with the predicted localization map to stylize the target object or scene based on sound information. The proposed INR can manipulate the localized pixel values to be semantically consistent with the provided audio input. Through a series of experiments, we show that the proposed framework outperforms the other audio-guided stylization methods. Moreover, LISA constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.
Learning Implicit Probability Distribution Functions for Symmetric Orientation Estimation from RGB Images Without Pose Labels
Venue: arXiv cs.CV 2022
Year: 2022

Object pose estimation is a necessary prerequisite for autonomous robotic manipulation, but the presence of symmetry increases the complexity of the pose estimation task. Existing methods for object pose estimation output a single 6D pose. Thus, they lack the ability to reason about symmetries. Lately, modeling object orientation as a non-parametric probability distribution on the SO(3) manifold by neural networks has shown impressive results. However, acquiring large-scale datasets to train pose estimation models remains a bottleneck. To address this limitation, we introduce an automatic pose labeling scheme. Given RGB-D images without object pose annotations and 3D object models, we design a two-stage pipeline consisting of point cloud registration and render-and-compare validation to generate multiple symmetrical pseudo-ground-truth pose labels for each image. Using the generated pose labels, we train an ImplicitPDF model to estimate the likelihood of an orientation hypothesis given an RGB image. An efficient hierarchical sampling of the SO(3) manifold enables tractable generation of the complete set of symmetries at multiple resolutions. During inference, the most likely orientation of the target object is estimated using gradient ascent. We evaluate the proposed automatic pose labeling scheme and the ImplicitPDF model on a photorealistic dataset and the T-Less dataset, demonstrating the advantages of the proposed method.
SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training
Venue: arXiv cs.CV 2022
Year: 2022

Video-language pre-training is crucial for learning powerful multi-modal representation. However, it typically requires a massive amount of computation. In this paper, we develop SMAUG, an efficient pre-training framework for video-language models. The foundation component in SMAUG is masked autoencoders. Different from prior works which only mask textual inputs, our masking strategy considers both visual and textual modalities, providing a better cross-modal alignment and saving more pre-training costs. On top of that, we introduce a space-time token sparsification module, which leverages context information to further select only "important" spatial regions and temporal frames for pre-training. Coupling all these designs allows our method to enjoy both competitive performances on text-to-video retrieval and video question answering tasks, and much less pre-training costs by 1.9X or more. For example, our SMAUG only needs about 50 NVIDIA A6000 GPU hours for pre-training to attain competitive performances on these two video-language tasks across six popular benchmarks.
Enhancing Crisis-Related Tweet Classification with Entity-Masked Language Modeling and Multi-Task Learning
Venue: arXiv cs.CL 2022
Year: 2022

Social media has become an important information source for crisis management and provides quick access to ongoing developments and critical information. However, classification models suffer from event-related biases and highly imbalanced label distributions which still poses a challenging task. To address these challenges, we propose a combination of entity-masked language modeling and hierarchical multi-label classification as a multi-task learning problem. We evaluate our method on tweets from the TREC-IS dataset and show an absolute performance gain w.r.t. F1-score of up to 10% for actionable information types. Moreover, we found that entity-masking reduces the effect of overfitting to in-domain events and enables improvements in cross-event generalization.