Reviewerly/exHarmony huggingface dataset:
Residual Hybrid Attention Network for Compression Artifact Reduction

Residual Hybrid Attention Network (RHAN) can restore images of arbitrary compression quality through flexibly fusing high-frequency features in the spatial and frequency domains based on the input quality factor. Specifically, to remove the compression artifacts, we propose a hybrid attention block (HAB) to adaptively restore the loss of high-frequency components, which parallelly predicts attention maps along two separate dimensions spatial and frequency spectra. To recover the compressed image flexibly and controllably, we further design a modulation decompression block (MDB), which employs a prior factor to learn a pair of modulation parameters and performs adaptively affine transformation on the obtained high-frequency features, thereby achieving high-quality image restoration at arbitrary compression levels. The quantitative and qualitative experiments on various public data sets show that the RHAN achieves the best performance and optimal visual perceptual quality in the JPEG image restoration with arbitrary compression levels.
The (legal) datafication of open science

The process of datafication of many aspects of our society, of the economy, and of the creative and scientific process has emerged as a distinctive element of this decade. This keynote will illustrate how datafication has translated into legal contentions in open science.
Learning from the Raw Domain: Cross Modality Distillation for Compressed Video Action Recognition

Video action recognition is faced with the challenges of both huge computation burden and performance requirements. Using compressed domain data, which saves much decoding computation, is a possible solution. Unfortunately, existing compressed-domain-based (CD) methods fail to obtain high performance, compared with state-of-the-art (SOTA) raw-domain-based (RD) methods. In order to solve the problem, we propose a cross-modality knowledge distillation method to force the CD model to learn the knowledge from the RD model. In particular, spatial knowledge and temporal knowledge are first constructed to align feature space between the raw domain and the compressed domain. Then, an adaptively multi-path knowledge learning scheme is presented to help the CD model learn in a more efficient way. Experiments verify the effectiveness of the proposed method in large-scale and small-scale datasets.
Self-Supervised Learning with Bi-Label Masked Speech Prediction for Streaming Multi-Talker Speech Recognition

Self-supervised learning (SSL), which utilizes the input data itself for representation learning, has achieved state-of-the-art results for various downstream speech tasks. However, most of the previous studies focused on offline single-talker applications, with limited investigations in multi-talker cases, especially for streaming scenarios. In this paper, we investigate SSL for streaming multi-talker speech recognition, which generates transcriptions of overlapping speakers in a streaming fashion. Firstly, we observe that conventional SSL techniques do not work well on this task due to the poor representation of overlapping speech. We then propose a novel SSL training objective, referred to as bi-label masked speech prediction, which explicitly preserves representations of all speakers in overlapping speech. We investigate various aspects of the proposed system, including data configuration and quantizer selection. The proposed SSL setup achieves substantially better word error rates on the LibriSpeechMix dataset.
HTNet: Human Topology aware network for 3d Human pose estimation

3D human pose estimation errors would propagate along the human body topology and accumulate at the end joints of limbs. Inspired by the backtracking mechanism in automatic control systems, we design an Intra-Part Constraint module that utilizes the parent nodes as the reference to build topological constraints for end joints at the part level. Further considering the hierarchy of the human topology, joint-level and body-level dependencies are captured via graph convolutional networks and self-attentions, respectively. Based on these designs, we propose a novel Human Topology aware Network (HTNet), which adopts a channel-split progressive strategy to sequentially learn the structural priors of the human topology from multiple semantic levels: joint, part, and body. Extensive experiments show that the proposed method improves the estimation accuracy by 18.7% on the end joints of limbs and achieves state-of-the-art results on Human3.6M and MPI-INF-3DHP datasets. Code is available at https://github.com/vefalun/HTNet.
STYX: Adaptive Poisoning Attacks Against Byzantine-Robust Defenses in Federated Learning

Decentralized training of machine learning models, for instance with federated learning protocols, continues to diffuse from theory toward practical applications and use cases. In federated learning (FL), a central server trains a model collaboratively with a group of users by communicating model updates, without the exchange of private user information. However, these systems can be influenced during training by malicious users who send poisoned updates. Because the training is decentralized and each user controls their own device, these users are free to poison the training protocol. In turn, this has lead to a number of proposals to incorporate aggregation strategies from byzantine-robust learning into the FL paradigm. Byzantine strategies are provably secure for simple model classes, and these robustness properties are often assumed to extend to neural models as well. In this work, we argue that a range of popular robust aggregation strategies, when applied to neural networks, can be trivially circumvented through simple adaptive attacks. We discuss the intuitions behind these adaptive attacks, and show that, despite their simplicity, they provide strong baselines that lead to significant decreases in model performance in FL systems.
Adaptive Submanifold-Preserving Sparse Regression for Feature Selection And Multiclass Classification

In this paper, we propose a novel embedded feature selection method, which is able to select the informative and discriminative features with the underlying submanifolds of data in intra-class being well preserved so as to improve the classification performance. Specifically, we first impose the l <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</inf> -norm on both loss function and projection matrix with the aim to suppress the influence of noise and achieve the sparse features selection. Then, a retargeted learning technique is introduced into our model to further enhance the discriminant ability of the projection matrix. Moreover, we design an adaptive graph regularizer to fully utilize the intra-class local submanifold information, which can endow the projection matrix with more locality preserving power and prevent overfitting. What is more, our method jointly optimizes problems of the projection learning, target learning and graph learning instead of separately learning them as in most of existing works, which guarantees an overall optimality in algorithmic performance. Experimental results on both synthetic and real-world image data sets demonstrate the superiorities of our method on local submanifold structure exploration and classification task.
Prototype-Based Layered Federated Cross-Modal Hashing

Recently, deep cross-modal hashing has gained increasing attention. However, in many practical cases, data are distributed and cannot be collected due to privacy concerns, which greatly reduces the cross-modal hashing performance on each client. And due to the problems of statistical heterogeneity, model heterogeneity, and forcing each client to accept the same parameters, applying federated learning to cross-modal hash learning becomes very tricky. In this paper, we propose a novel method called prototype-based layered federated cross-modal hashing. Specifically, the prototype is introduced to learn the similarity between instances and classes on server, reducing the impact of statistical heterogeneity (non-IID) on different clients. And we monitor the distance between local and global prototypes to further improve the performance. To realize personalized federated learning, a hypernetwork is deployed on server to dynamically update different layers’ weights of local model. Experimental results on benchmark datasets show that our method outperforms state-of-the-art methods.
Coarse-To-Fine Knowledge Selection for Document Grounded Dialogs

Multi-document grounded dialogue systems (DGDS) belong to a class of conversational agents that answer users’ requests by finding supporting knowledge from a collection of documents. Most previous studies aim to improve the knowledge retrieval model or propose more effective ways to incorporate external knowledge into a parametric generation model. These methods, however, focus on retrieving knowledge from mono-granularity language units (e.g. passages, sentences, or spans in documents), which is not enough to effectively and efficiently capture precise knowledge in long documents. This paper proposes Re3G, which aims to optimize both coarse-grained knowledge retrieval and fine-grained knowledge extraction in a unified framework. Specifically, the former efficiently finds relevant passages in a retrieval-and-reranking process, whereas the latter effectively extracts finer-grain spans within those passages to incorporate into a parametric answer generation model (BART, T5). Experiments on DialDoc Shared Task demonstrate the effectiveness of our method.
RANCANG BANGUN SISTEM INFORMASI PENGGAJIAN PADA BUILDING MANAGEMENTAPARTEMENT GRAND CENTERPOINT BEKASI

Payroll is very meaningful for companies or agencies so we need a system that facilitates payroll activities. Apartment Grand Centerpoint operates in the service sector in the form of flats, experiencing problems in processing employee attendance data due to the processing of attendance book data as a manual system, resulting in operational errors and irregularities in calculating and paying salaries. This research was conducted with the aim of designing a payroll system to facilitate accounting &amp; finance in payroll in companies with system features&#x0D; such as managing attendance data, managing employee data along with their positions, managing payroll data, managing payroll report data, journal reports, and employee payslips.
Prune Then Distill: Dataset Distillation with Importance Sampling

The development of large datasets for various tasks has driven the success of deep learning models but at the cost of increased label noise, duplication, collection challenges, storage capabilities, and training requirements. In this work, we investigate whether all samples in large datasets contribute equally to better model accuracy. We study statistical and mathematical techniques to reduce redundancies in datasets by directly optimizing data samples for the generalization accuracy of deep learning models. Existing dataset optimization approaches include analytic methods that remove unimportant samples and synthetic methods that generate new datasets to maximize the generalization accuracy. We develop Prune then distill, a combination of analytic and synthetic dataset optimization algorithms, and demonstrate up to 15% relative improvement in generalization accuracy over either approach used independently on standard image and audio classification tasks. Additionally, we demonstrate up to 38% improvement in generalization accuracy of dataset pruning algorithms by maintaining class balance while pruning.
Detection of Real-Time Deepfakes in Video Conferencing with Active Probing and Corneal Reflection

The COVID pandemic has led to the wide adoption of online video calls in recent years. However, the increasing reliance on video calls provides opportunities for new impersonation attacks by fraudsters using the advanced real-time DeepFakes. Real-time DeepFakes pose new challenges to detection methods, which have to run in real-time as a video call is ongoing. In this paper, we describe a new active forensic method to detect real-time DeepFakes. Specifically, we authenticate video calls by displaying a distinct pattern on the screen and using the corneal reflection extracted from the images of the call participant’s face. This pattern can be induced by a call participant displaying on a shared screen or directly integrated into the video-call client. In either case, no specialized imaging or lighting hardware is required. Through large-scale simulations, we evaluate the reliability of this approach under a range in a variety of real-world imaging scenarios.
Stargan-vc Based Cross-Domain Data Augmentation for Speaker Verification

Automatic speaker verification (ASV) faces domain shift caused by the mismatch of intrinsic and extrinsic factors, such as recording device and speaking style, in real-world applications, which leads to severe performance degradation. Since single-speaker multi-condition (SSMC) data is difficult to collect in practice, existing domain adaptation methods are hard to ensure the feature consistency of the same class but different domains. To this end, we propose a cross-domain data generation method to obtain a domain-invariant ASV system. Inspired by voice conversion (VC) task, a StarGAN based generative model first learns cross-domain mappings from SSMC data, and then generates missing domain data for all speakers, thus increasing the intra-class diversity of the training set. Considering the difference between ASV and VC task, we renovate the corresponding training objectives and network structure to make the adaptation task-specific. Evaluations on achieve a relative performance improvement of about 5-8% over the baseline in terms of minDCF and EER, outperforming the CNSRC winner’s system of the equivalent scale.
Rethinking Rule-Based Approaches in Session-Based Recommendation

Existing approaches to session-based recommendation are focused on using advanced deep neural networks (DNNs). Recent studies have shown that some traditional methods can outperform some DNN-based models. However, in recent years, few studies have tried to build traditional models. In this paper, we investigate this issue and propose a concise rule-based method for session-based recommendation. Specifically, we make item adjacent and N-gram-based rules to extract frequenting, sequential and other patterns of the limited historical information to construct item correlation dictionaries. Then, we exploit these dictionaries at the inference stage by constructing candidate item generation and item set fusion rules to acquire candidate items. By this means, we can leverage short-term user-item interaction records to generate candidate items for anonymous sessions. Finally, we sort the candidate items to make a recommendation. Extensive experimental results on three real public datasets show that our method can significantly outperform existing traditional methods and even outperforms several representative DNN-based models. The model just costs a few seconds to train and inference, and occupies less memory space than that. Compared with the previous models, it has a huge lead in time and space.
An End-to-End Framework for Partial View-Aligned Clustering with Graph Structure

Over the last decade, many multi-view clustering (MVC) methods have achieved promising results with intact and completely correct correspondence of multi-view data, which is hard to satisfy in practice leading to the problem of partially view-aligned clustering. In this paper, we propose a novel method to tackle it, termed An End-to-end Framework for Partial View-aligned Clustering with Graph structure(EGPVC). It employs Dykstra’s cyclic constraint projection algorithm to obtain the correspondence between two views. In particular, EGPVC develops an end-to-end framework for partially view-aligned clustering, in which representation learning and clustering process can benefit from each other through the deep embedded clustering layer. Moreover, a cross-view graph regularization term is designed to improve the quality of the learned common representation with graph structure information. Experimental results on several real-world datasets show our promising results comparing with the state-of-the-art methods in partially view-aligned clustering.
Parallel Sentence-Level Explanation Generation for Real-World Low-Resource Scenarios

In order to reveal the rationale behind model predictions, many works have exploited providing explanations in various forms. Recently, to further guarantee readability, more and more works turn to generate sentence-level human language explanations. However, current works pursuing sentence- level explanations rely heavily on annotated training data, which limits the development of interpretability to only a few tasks. As far as we know, this paper is the first to explore this problem smoothly from weak-supervised learning to unsupervised learning. Besides, we also notice the high latency of autoregressive sentence-level explanation generation, which leads to asynchronous interpretability after prediction. Therefore, we propose a non-autoregressive interpretable model to facilitate parallel explanation generation and simultaneous prediction. Through extensive experiments on Natural Language Inference task and Spouse Prediction task, we find that users are able to train classifiers with comparable performance 10 − 15× faster with parallel explanation generation using only a few or no annotated training data.
Training Set Cleansing of Backdoor Poisoning by Self-Supervised Representation Learning

A backdoor or Trojan attack is an important type of data poisoning attack against deep neural network (DNN) classifiers, wherein the training dataset is poisoned with a small number of samples that each possess the backdoor pattern (usually a pattern that is either imperceptible or innocuous) and which are mislabeled to the attacker’s target class. When trained on a backdoor-poisoned dataset, a DNN behaves normally on most benign test samples but makes incorrect predictions to the target class when the test sample has the backdoor pattern incorporated (i.e., contains a backdoor trigger). Here we focus on image classification tasks and show that supervised training may build stronger association between the backdoor pattern and the associated target class than that between normal features and the true class of origin. By contrast, self-supervised representation learning ignores the labels of samples and learns a feature embedding based on images’ semantic content. Using a feature embedding found by self-supervised representation learning, a data cleansing method, which combines sample filtering and relabeling, is developed. Experiments on CIFAR-10 benchmark datasets show that our method achieves state-of-the-art performance in mitigating backdoor attacks.
Adversarial Network Pruning by Filter Robustness Estimation

Network pruning has been extensively studied in model compression to reduce neural networks’ memory, latency, and computation cost. However, the pruned networks still suffer from the threat posed by adversarial examples, limiting the broader application of the pruned networks in safety-critical applications. Previous studies maintain the robustness of the pruned networks by combining adversarial training and network pruning but ignore preserving the robustness at a high sparsity ratio in structured pruning. To address such a problem, we propose an effective filter importance criterion, Filter Robustness Estimation (FRE), to evaluate the importance of filters by estimating their contribution to the adversarial training loss. Empirical results show that our FRE-based Robustness-aware Filter Pruning (FRFP) outperforms the state-of-the-art methods by 12.19%∼37.01% of empirical robust accuracy on the CIFAR10 dataset with the VGG16 network at an extreme pruning ratio of 90%.
Learning ASR Pathways: A Sparse Multilingual ASR Model

Neural network pruning compresses automatic speech recognition (ASR) models effectively. However, in multilingual ASR, language-agnostic pruning may lead to severe performance drops on some languages because language-agnostic pruning masks may not fit all languages and discard important language-specific parameters. In this work, we present ASR pathways, a sparse multilingual ASR model that activates language-specific sub-networks ("pathways"), such that the parameters for each language are learned explicitly. With the overlapping sub-networks, the shared parameters can also enable knowledge transfer for lower-resource languages via joint multilingual training. We propose a novel algorithm to learn ASR pathways, and evaluate the proposed method on 4 languages with a streaming RNN-T model. Our proposed ASR pathways outperform both dense models and a language-agnostically pruned model, and provide better performance on low-resource languages compared to the monolingual sparse models.
Towards a Unified Training for Levenshtein Transformer

Levenshtein Transformer (LevT) is a widely-used text-editing model, which generates a sequence based on editing operations (deletion and insertion) in a non-autoregressive manner. However, it is challenging to train the key refinement components of LevT due to training-inference discrepancy. By carefully designing experiments, our work reveals that the deletion module is under-trained while the insertion module is over-trained due to the imbalance training signals for the two refinement modules. Based on these observations, we further propose a dual learning approach that can remedy the imbalance training by feeding an initial input to both refinement modules, which is consistent with the process in inference. Experimental results on three representative NLP tasks demonstrate the effectiveness and universality of the proposed approach. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Multi-Head Uncertainty Inference for Adversarial Attack Detection

Deep neural networks (DNNs) are sensitive and susceptible to tiny perturbations by adversarial attacks which cause erroneous predictions. Various methods, including adversarial defense and uncertainty inference (UI), have been developed to overcome adversarial attacks in recent years. In this paper, we propose a multi-head uncertainty inference (MH-UI) framework for detecting adversarial attack examples. We adopt a multi-head architecture with multiple prediction heads (i.e., classifiers) to obtain predictions from different depths in the DNNs and introduce shallow information for the UI. Using independent heads at different depths, the normalized predictions are assumed to follow the same Dirichlet distribution, and we estimate the distribution parameter of it by moment matching. Cognitive uncertainty brought by the adversarial attacks will be reflected and amplified in the distribution. Experimental results show that the proposed MH-UI framework has good performance in different settings of adversarial attack detection tasks.
Estimation of Visual Contents from Human Brain Signals via VQA Based on Brain-Specific Attention

This paper presents a method for estimation of visual cognitive contents from human brain signals via a newly derived visual question answering (VQA) model. The proposed method can estimate a wide range of cognitive contents from functional magnetic resonance imaging data when subjects viewed images via the VQA model which can effectively utilize low-level and high-level image features extracted from the viewed images. To make use of multiple image features, we newly introduce brain-specific attention into the VQA model. The brain-specific attention can adaptively determine the significance of image features at each level depending on the complexity of the cognitive contents (e.g., category, pattern, number and color). Therefore, we enable flexible and extensive estimation of various cognitive contents. Experimental results show that the proposed method can significantly improve the performance of the visual content estimation.