Reviewerly/exHarmony huggingface dataset:
Underwater Image Restoration with Light-Aware Progressive Network

Underwater images suffer from different types of degradation such as blur and color cast. It is difficult to obtain images with both color equalization and rich texture in various underwater scenes. To alleviate these issues, this paper proposes a reflected light-aware multi-scale progressive restoration network. A restoration framework is designed based on physical imaging model. It can accurately remove scattering blur of various underwater images and output a preliminary restoration result. A multi-scale progressive enhancement module is proposed to further enrich the image details and improve the generalization ability of the model by fully extracting detailed and spatial features. For the issues of color cast and low contrast, a new chromatic aberration correction mechanism is introduced, which makes the image color balanced. The experimental results show that our method can obtain underwater images with enriching texture and harmonious color distribution as compared with other state-of-the-art methods.
Smoothing Point Adjustment-Based Evaluation of Time Series Anomaly Detection

Anomalies in time series appear consecutively, forming anomaly segments. Applying the classical point-based evaluation metrics to evaluate the detection performance of segments leads to considerable underestimation, so most related studies resort to point adjustment. This operation treats all points as true positives within a segment equally when only one individual point alarms, resulting in significant overestimation and creating an illusion of superior performance. This paper proposes smoothing point adjustment, a novel range-based evaluation protocol for time series anomaly detection. Our protocol reflects detection performance impartially by carefully considering the specific location and frequency of alarms in the raw results. It is achieved by smoothly determining the adjustment range and rewarding early detection via a ranging function and a rewarding function. Compared with other evaluation metrics, experiments on different datasets show that our protocol can yield a performance ranking of various methods more consistent with the desired situation.
Research on the Implementation of Advertising Design Teaching Based on Unity3D Development Platform and Web3D Technology

In this work, the Unity3D development platform and Web3D technology are integrated into the teaching method of advertising design to get rid of the issues due to lack of communication and efficacy through design of digital advertisement. Based on this, an approach for ingenious product design from nature is proposed, with an emphasis on attaining a functional interaction of aesthetic intent and geometric features and investigating the relationships amongst natural systems and designers in product design from nature (PDN). The ponderings and research findings for the methodologies associated with the proposed approach have been presented. This methodology is thought to significantly bring down the delivery time of ground-breaking design and development of products, both economically and technologically. The findings in comprehensive experiments demonstrates that interactive virtual technology can significantly enhance the efficacy and interaction of the whole system in the process of digital advertising design.
Improving Image Captioning with Control Signal of Sentence Quality

In the dataset of image captioning, each image is aligned with several descriptions. Despite the fact that the quality of these descriptions varies, existing captioning models treat them equally in the training process. In this paper, we propose a new control signal of sentence quality, which is taken as an additional input to the captioning model. By integrating the control signal information, captioning models are aware of the quality level of the target sentences and handle them differently. Moreover, we propose a novel reinforcement training method specially designed for the control signal of sentence quality: Quality-oriented Self-Annotated Training (Q-SAT). Extensive experiments on MSCOCO dataset show that without extra information from ground truth captions, models controlled by the highest quality level outperform baseline models on accuracy-based evaluation metrics, which validates the effectiveness of our proposed methods.
Adapter Tuning With Task-Aware Attention Mechanism

Adapter-tuning inserts simple feed-forward layers (adapters) in pre-trained language models (PLMs) and just tunes the adapters when transferring to downstream tasks, having become the state-of-the-art parameter-efficient tuning (PET) strategy. Although the adapters aim to learn task-related representations, their inputs are still obtained from the task-independent and frozen multi-head attention (MHA) modules, leading to insufficient utilization of contextual information for various downstream tasks. Intuitively, MHA should be task-dependent and could attend to different contexts in different downstream tasks. Thus, this paper proposes the task-aware attention mechanism (TAM) to enhance adapter tuning. Specifically, we first utilize the task-dependent adapter to generate token-wise task embedding. Then, we apply the task embedding to influence MHA which task-dependently aggregates the contextual information. Experimental results on a wide range of natural language understanding and generation tasks demonstrate the effectiveness of our method. Furthermore, extensive analyses demonstrate that the generated task embedding corresponds with the difficulty of tasks.
Recouple Event Field via Probabilistic Bias for Event Extraction

Event Extraction (EE), aiming to identify and classify event triggers and arguments from event mentions, has benefited from pre-trained language models (PLMs). However, existing PLM-based methods ignore the information of trigger/argument fields, which is crucial for understanding event schemas. To this end, we propose a Probabilistic reCoupling model enhanced Event extraction framework (ProCE). Specifically, we first model the syntactic-related event fields as probabilistic biases, to clarify the event fields from ambiguous entanglement. Furthermore, considering multiple occurrences of the same triggers/arguments in EE, we explore probabilistic interaction strategies among multiple fields of the same triggers/arguments, to recouple the corresponding clarified distributions and capture more latent information fields. Experiments on EE datasets demonstrate the effectiveness and generalization of our proposed approach.
Semi-Supervised Local Structured Feature Learning with Dynamic Maximum Entropy Graph

In this paper, we propose a novel semi-supervised dimensionality reduction method based on local structured feature learning with dynamic maximum entropy graph. The proposed method first learns a local discriminative embedding subspace from labeled data for preserving the intrinsic sub-manifold structure in each class by virtue of dynamic maximum entropy graph technique. Then, another auto-optimized k-nearest neighbor graph is constructed at learned embedded subspace to smooth the manifold of all labeled and unlabeled data such that each labeled sample and its neighbored unlabeled samples can be clustered into a same sub-manifold and possess the same label information. Most importantly, in order to guarantee an overall optimum, subspace learning and local structure graph optimizing are performed simultaneously rather than treat them as two irrelevant steps as done in most of graph-based semi-supervised learning methods, which can avoid the negative effects brought by noisy and redundant features. Experimental results on several real-world benchmarks demonstrate the superiorities of our method on sub-manifold structure exploration and classification task.
MSNet: A Deep Architecture Using Multi-Sentiment Semantics for Sentiment-Aware Image Style Transfer

Sentiment plays an essential role in people’s perception of images. To incorporate the sentiment information into the image style transfer task for better sentiment-aware performance, we introduce a new task named sentiment-aware image style transfer. To solve this problem, we first introduce a novel Multi-Sentiment Semantics Space (MSS-Space) to capture the non-deterministic and complicated nature of sentiment semantics. With the MSS-Space, we establish tight associations between the visual attributes of images and the multi-sentiment semantics by minimizing their distance in MSS-Space and then propose the Multi-Sentiment Style Transfer Net (MSNet). Experiments demonstrate that, compared with three competing models, our proposed MSNet generates more explicit images and better preserves the integrity of salient objects, local details, and multi-sentiment. In particular, our model outperforms the state-of-the-art by +28.72% in terms of the top-3 accuracy on average.
Enhancing Multimodal Alignment with Momentum Augmentation for Dense Video Captioning

Dense video captioning aims to localize multiple events from an untrimmed video and generate corresponding captions for each event. Fusing different modalities(e.g. rgb, flow, audio) via transformer structure is a promising way to improve the caption performance. However, it is challenging for the cross-modal encoder to learn multimodal interactions due to their inherent disparities of distribution. In this paper, we propose a novel transformer structure with contrastive learning to align different modalities. Specifically, to avoid the limitation of small batch size and false contrastive targets, we design an event-aligned momentum augmentation strategy to apply contrast learning for dense video captioning. The experimental result shows that our proposals outperform all existing multimodal fusion methods for dense video captioning.
Web Page Content Block Identification with Extended Block Properties

Web page segmentation is one of the most influential factors for the automated integration of web page content with other systems. Existing solutions are focused on segmentation but do not provide a more detailed description of the segment including its range (minimum and maximum HTML code bounds, covering the segment content) and variants (the same segments with different content). Therefore the paper proposes a novel solution designed to find all web page content blocks and detail them for further usage. It applies text similarity and document object model (DOM) tree analysis methods to indicate the maximum and minimum ranges of each identified HTML block. In addition, it indicates its relation to other blocks, including hierarchical as well as sibling blocks. The evaluation of the method reveals its ability to identify more content blocks in comparison to human labeling (in manual labeling only 24% of blocks were labeled). By using the proposed method, manual labeling effort could be reduced by at least 70%. Better performance was observed in comparison to other analyzed web page segmentation methods, and better recall was achieved due to focus on processing every block present on a page, and providing a more detailed web page division into content block data by presenting block boundary range and block variation data.
LiteG2P: A Fast, Light and High Accuracy Model for Grapheme-to-Phoneme Conversion

As a key component of automated speech recognition (ASR) and the front-end in text-to-speech (TTS), grapheme-to-phoneme (G2P) plays the role of converting letters to their corresponding pronunciations. Existing methods are either slow or poor in performance, and are limited in application scenarios, particularly in the process of on-device inference. In this paper, we integrate the advantages of both expert knowledge and connectionist temporal classification (CTC) based neural network and propose a novel method named LiteG2P which is fast, light and theoretically parallel. With the carefully leading design, LiteG2P can be applied both on cloud and on device. Experimental results on the CMU dataset show that the performance of the proposed method is superior to the state-of-the-art CTC based method with 10 times fewer parameters, and even comparable to the state-of-the-art Transformer-based sequence-to-sequence model with less parameters and 33 times less computation.
Tracking Targets in Hyper-Scale Cameras Using Movement Predication

Hyper-scale surveillance cameras have enabled seamless target (e.g., vehicles, individuals) tracking in urban scenarios, significantly improving everyday safety and emergency response capacity. However, practically tracking multiple targets among cameras would incur prohibitive huge computation costs, even with the acceleration utility of edge computing. This work observes with real-world camera deployment that existing tracking scheduling is unfortunately inefficient due to the redundant and over-activation of cameras. We then propose the design of a hierarchical tracking framework HyMOT that exploits fine-grained target movement prediction for efficient tracking in hyper-scale cameras. At its core, HyMOT builds a probabilistic target movement graph with tempo-spatial correlation knowledge extracted from historical statistics. With the graph, we formulate tracking scheduling as an optimization problem with efficiency-accuracy tradeoff constraints and solve this NP-hard problem with a greedy strategy. Experiments based on the Cityflow and Geolife datasets demonstrate that, compared with two baselines, Hy-MOT requires significantly less (over 90%) computation workloads to track the same amount of targets with similar accuracy.
Optimal Transport with a Diversified Memory Bank for Cross-Domain Speaker Verification

Optimal transport (OT) can be applied to cross-domain adaptation in speaker verification (SV) by converting speakers' probability distributions from source to target domains. However, in scenarios involving over-massive categories (speakers) or difficult samples in discrimination, OT often has difficulty computing effective transports. To address this challenge, we propose an OT-based unsupervised domain adaptation (UDA) framework for SV, OT with a diversified memory bank, called DMB-OT, which ensures the accuracy of transfers by two strategies: (1) It regularizes the solution space of OT, which attempts to plan transformations between audio samples from the same speaker with high confidence; (2) it integrates a dynamic curriculum learning algorithm, preventing OT from calculating transport couplings based on hard-discriminative samples in the early stage of UDA. Experiments under different target domains showed that our unsupervised DMB-OT could significantly improve the performance of OT-based UDA and could even match the performance of the supervised PLDA-based adaptation.
Meta-Dag: Meta Causal Discovery Via Bilevel Optimization

Causal discovery in the form of directed acyclic graph (DAG) structure learning finds causal relationships among features of sampled data and has been recognized as one of the most important problems in causal inference. Recent works show that DAGs can be learned by solving a continuous optimization problem with a functional equality constraint. However, popular causal models generally assume that the data are i.i.d. in the sense that all data samples are generated by only one underlying causal graph. In this work, we propose a general causal learning model inspired by meta-learning, which aims at finding an invariant DAG over multiple domains and increasing the generalization performance of DAG structure discovery. Mathematically, this model is formulated as a functional constrained bilevel optimization problem that can be solved by our proposed bilevel primal-dual (BPD) algorithm with provable convergence rate guarantees. Numerous numerical experiments demonstrate that the proposed meta-DAG model and BPD algorithm outperform the benchmarks in terms of reconstruction errors and graph Hamming distance.
Overcoming the Seesaw in Monocular 3D Object Detection Via Language Knowledge Transferring

Monocular 3D object detection is a challenging problem in self-driving and computer vision communities. Previous works suffered from a severe seesaw phenomenon: multi-category learning was worse than single-category, and feature learning between categories inhibited each other. We reveal that the real culprit is the significant difference in depth distribution between categories. Confusing feature representations exacerbate depth estimation. In this paper, we propose Language Knowledge Transferring to introduce language information in monocular 3D object detection, termed as MonoLT. Multimodal language-Image guides networks learn more class-specific features, which reduces the pressure of depth estimation. Meanwhile, we propose the Polar Depth Aggregator to make the depth estimation less disturbed by the environment and other instances (especially different classes). Comprehensive experiments performed on the KITTI dataset prove the superiority of our proposed method. The code will be released soon.
Self-Supervised Learning with Explorative Knowledge Distillation

Previous paradigms have combined self-supervised learning (SSL) with knowledge distillation to compress a self-supervised teacher model into a smaller student. In this work, we devise a self-supervised explorative distillation (SSED) algorithm to improve the representation quality of the lightweight models. We introduce a heterogeneous teacher to maximumly learn rich feature representation for the student, which reaches the expected goal of capturing discriminative feature information contained in network itself. SSED enforces the student to learn more diversified and perfect representations of the original class recognition task and self-supervised learning task. Extensive experiments show that SSED improves accuracy effectively on large and small models, and surpassing current top-performing SSL methods. Particularly, the linear results of our ResNet-18, trained with ResNet-50 teacher, achieves 65.5% ImageNet top-1 accuracy, which is 1.4% and 4.9% higher than OSS and DisCo. Code is available at https://github.com/nanxiaotong/SSED.
Kernel Ridge Regression for Generalized Graph Signal Processing

In generalized graph signal processing (GGSP), a function (an element from a separable Hilbert space) is associated with each vertex. To perform non-linear filtering and regression under the GGSP framework, we formulate an operator-valued kernel ridge regression (KRR) filtering approach. Under a specific choice of separable kernels, we show that this problem is equivalent to learning a nonlinear frequency response on each frequency band. We specify the choice of the reproducing kernel according to the signal’s spectral properties and discuss its effect on the learning result. The proposed approach is validated on a real dataset and demonstrated to outperform other competing methods.
Nested Attention Network with Graph Filtering for Visual Question and Answering

Recently, Visual Question Answering(VQA), which is required to generate the answer by understanding both visual and textual content, has attracted considerable research interest. Most existing works extract visual features with the CNN network and learn its feature embedding with an attention mechanism. However, this mechanism may ignore the interaction between entities in the image, which has a fuzzy impact on the answer generation. To better explore the relationship between different entities in the image, a novel Nested Attention Network with Graph Filtering (NANGF) is proposed. It composes of two novel designed modules: a graph filtering mechanism to mine more precise visual semantics and avoid understanding deviation and nested attention to effectively guide the integration of visual features and question features. Extensive experiments conducted on the VQA2.0 datasets demonstrate the effectiveness of the proposed method.
T5lephone: Bridging Speech and Text Self-Supervised Models for Spoken Language Understanding Via Phoneme Level T5

In Spoken language understanding (SLU), a natural solution is concatenating pre-trained speech models (e.g. HuBERT) and pretrained language models (PLM, e.g. T5). Most previous works use pre-trained language models with subword-based tokenization. However, the granularity of input units affects the alignment of speech model outputs and language model inputs, and PLM with character-based tokenization is underexplored. In this work, we conduct extensive studies on how PLMs with different tokenization strategies affect spoken language understanding task including spoken question answering (SQA) and speech translation (ST).We further extend the idea to create T5lephone <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , a variant of T5 that is pretrained using phonemicized text. We initialize T5lephone with existing PLMs to pretrain it using relatively lightweight computational resources. We reached state-of-the-art on NMSQA, and the T5lephone model exceeds T5 with other types of units on end-to-end SQA and ST. Our code is publicly available. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>
Local to global prior Learning for blind Unsupervised Image super Resolution

Deep convolutional neural networks (DCNN) have dominated the single image super resolution (SR) field, and demonstrated significant success in generating high-resolution (HR) images from the ideal low-resolution (LR) images captured under controlled imaging conditions. However, the resolved image performance would be dramatically degraded for the LR image captured in real environment. Recently, some works attempt to automatically learn image and kernel priors by leveraging the local statistic modeling capability of the DCNNs from the LR observation only, and illustrated the feasibility to construct a specific CNN for real-world application. The basic convolution operation in the DCNNs can potentially capture the local interaction to generate plausible image appearance but are far from sufficiency to achieve global context, which has been proven to be a promising property of a transformer block, to further boost the SR performance. This study proposes a cooperative local to global prior learning (LoGPT) framework for blind unsupervised image super resolution by jointly modeling the local connectivity with convolution operations and global context with transformer block. Specifically, we elaborate a multi-scale encoder-decoder architecture configuring with the convolution blocks on the high resolution scales to learn local priors while the transformer block on the low-resolution scales to capture long-range dependencies, and then incorporate with a simple convolution-based subnet to simultaneously learning the local to global image priors and kernel priors in an unsupervised way using the observed LR image only. Extensive experiments have demonstrated that our proposed blind SR method achieves superior SR performance over both supervised and unsupervised state-of-the-art methods in term of quantitative metrics and perceptive quality.