Reviewerly/exHarmony huggingface dataset:
Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution

Deep convolutional neural networks (CNNs) have pushed forward the frontier of super-resolution (SR) research. However, current CNN models exhibit a major flaw: they are biased towards learning low-frequency signals. This bias becomes more problematic for the image SR task which targets reconstructing all fine details and image textures. To tackle this challenge, we propose to improve the learning of high-frequency features both locally and globally and introduce two novel architectural units to existing SR models. Specifically, we propose a dynamic highpass filtering (HPF) module that locally applies adaptive filter weights for each spatial location and channel group to preserve high-frequency signals. We also propose a matrix multi-spectral channel attention (MMCA) module that predicts the attention map of features decomposed in the frequency domain. This module operates in a global context to adaptively recalibrate feature responses at different frequencies. Extensive qualitative and quantitative results demonstrate that our proposed modules achieve better accuracy and visual improvements against state-of-the-art methods on several benchmark datasets.
Seeking Similarities over Differences: Similarity-based Domain Alignment for Adaptive Object Detection

In order to robustly deploy object detectors across a wide range of scenarios, they should be adaptable to shifts in the input distribution without the need to constantly an-notate new data. This has motivated research in Unsupervised Domain Adaptation (UDA) algorithms for detection. UDA methods learn to adapt from labeled source domains to unlabeled target domains, by inducing alignment between detector features from source and target domains. Yet, there is no consensus on what features to align and how to do the alignment. In our work, we propose a framework that generalizes the different components commonly used by UDA methods laying the ground for an in-depth analysis of the UDA design space. Specifically, we propose a novel UDA algorithm, ViSGA, a direct implementation of our framework, that leverages the best design choices and introduces a simple but effective method to aggregate features at instance-level based on visual similarity before inducing group alignment via adversarial training. We show that both similarity-based grouping and adversarial training allows our model to focus on coarsely aligning feature groups, without being forced to match all instances across loosely aligned domains. Finally, we examine the applicability of ViSGA to the setting where labeled data are gathered from different sources. Experiments show that not only our method outperforms previous single-source approaches on Sim2Real and Adverse Weather, but also generalizes well to the multi-source setting.
Exploring Inter-Channel Correlation for Diversity-preserved Knowledge Distillation

Knowledge Distillation has shown very promising ability in transferring learned representation from the larger model (teacher) to the smaller one (student). Despite many efforts, prior methods ignore the important role of retaining inter-channel correlation of features, leading to the lack of capturing intrinsic distribution of the feature space and sufficient diversity properties of features in the teacher network. To solve the issue, we propose the novel Inter-Channel Correlation for Knowledge Distillation (ICKD), with which the diversity and homology of the feature space of the student network can align with that of the teacher network. The correlation between these two channels is interpreted as diversity if they are irrelevant to each other, otherwise homology. Then the student is required to mimic the correlation within its own embedding space. In addition, we introduce the grid-level inter-channel correlation, making it capable of dense prediction tasks. Extensive experiments on two vision tasks, including ImageNet classification and Pascal VOC segmentation, demonstrate the superiority of our ICKD, which consistently outperforms many existing methods, advancing the state-of-the-art in the fields of Knowledge Distillation. To our knowledge, we are the first method based on knowledge distillation boosts ResNet18 beyond 72% Top-1 accuracy on ImageNet classification. Code is available at: https://github.com/ADLab-AutoDrive/ICKD.
Combating emerging financial risks in the big data era: A perspective review

Big data technology has had a significant impact on new business and financial services: for example, GPS and Bluetooth inspire location-based services, and search and web technologies motivate online shopping, reviews, and payments. These business services have become more connected than ever, and as a result, financial frauds have become a significant challenge. Therefore, combating financial risks in the big data era requires breaking the borders of traditional data, algorithms, and systems. An increasing number of studies have addressed these challenges and proposed new methods for risk detection, assessment, and forecasting. As a key contribution, we categorize these works in a rational framework: first, we identify the data that can be used to identify risks. We then discuss how big data can be combined with the emerging tools to effectively learn or analyze financial risk. Finally, we highlight the effectiveness of these methods in real-world applications. Furthermore, we stress on the importance of utilizing multi-channel information, graphs, and networks of long-range dependence for the effective identification of financial risks. We conclude our survey with a discussion on the new challenges faced by the financial sector, namely, deep fake technology, adversaries, causal and interpretable inference, privacy protection, and microsimulations.
Zeroing Neural Network With Fuzzy Parameter for Computing Pseudoinverse of Arbitrary Matrix

A correlation between fuzzy logic systems (FLS) and zeroing neural networks (ZNN) design is investigated. It is shown that the gain parameter included in ZNN design can be dynamically adjusted over time by means of an appropriate value derived as the output of a properly defined FLS, which includes appropriately defined membership functions and fuzzy logic rules. Dynamical systems which are applicable to time-varying rank-deficient matrices are proposed. Convergence properties are investigated and illustrative simulation experiments are performed. Presented simulation experiments confirm the superiority of the FLS proposed in this article with respect to previously proposed FLS for dynamic adjustment of gain parameters. Furthermore, the superiority of the FLS-based ZNN model over the corresponding ZNN models based on the classical approach in defining the varying-gain parameter is demonstrated.
Scrambled Linear Pseudorandom Number Generators

F 2 -linear pseudorandom number generators are very popular due to their high speed, to the ease with which generators with a sizable state space can be created, and to their provable theoretical properties. However, they suffer from linear artifacts that show as failures in linearity-related statistical tests such as the binary-rank and the linear-complexity test. In this article, we give two new contributions. First, we introduce two new F 2 -linear transformations that have been handcrafted to have good statistical properties and at the same time to be programmable very efficiently on superscalar processors, or even directly in hardware. Then, we describe some scramblers , that is, nonlinear functions applied to the state array that reduce or delete the linear artifacts, and propose combinations of linear transformations and scramblers that give extremely fast pseudorandom number generators of high quality. A novelty in our approach is that we use ideas from the theory of filtered linear-feedback shift registers to prove some properties of our scramblers, rather than relying purely on heuristics. In the end, we provide simple, extremely fast generators that use a few hundred bits of memory, have provable properties, and pass strong statistical tests.
Adaptive Privacy-Preserving Federated Learning for Fault Diagnosis in Internet of Ships

The recent appearance of Internet of Things (IoT) technologies applied in the maritime industry has introduced the Internet of Ships (IoS) paradigm. By leveraging IoS and deep learning (DL), various DL-based fault diagnosis methods have been proposed to improve shipping companies’ maintenance performance and reduce operational costs. However, the traditional centralized learning approach (CL), which centralizes the data resources of different shipping companies to a cloud server for model training, is restricted in real industrial scenarios due to privacy concerns and business competitions. In this article, we propose a novel adaptive privacy-preserving federated learning approach, named AdaPFL, for fault diagnosis in IoS, which can organize different shipping agents to collaboratively develop a model by sharing model parameters with no risk of data leakage. First, we use two common tasks as examples to demonstrate that a small part of the model parameters might reveal the shipping agents’ raw information. Based on this, the Paillier-based communication scheme is designed to preserve the raw information of the shipping agents. Furthermore, to deal with the harsh marine environment, a control algorithm is proposed to adaptively change the model aggregation interval during the training process for reducing cryptography computation and communication costs. Theoretical analysis and experiments prove the high effectiveness of the AdaPFL on a real nonindependent and identically distributed (non-i.i.d) fault data set.
Matching Sensor Ontologies With Multi-Context Similarity Measure and Parallel Compact Differential Evolution Algorithm

Sensor ontology is able to resolve the sensor data heterogeneity issue among the Cybertwin-driven 6G based Internet of Everything (IoE) systems. However, due to human subjectivity, the sensor ontologies also suffer from the heterogeneity problem. To address this problem, it is necessary to execute the sensor Ontology Matching (OM) process, i.e., finding the identical entity pairs between two ontologies. To this end, we first propose a Multi-Context based ESM (MC-ESM) to measure the similarity of two entities by taking into consideration their semantic contexts. After that, a parallel compact Differential Evolution with Adaptive Step Length (pcDE-ASL) is proposed to find all entity mappings, which uses the probability presentation on the population to save the memory consumption, and the parallel processing mechanism and ASL to help the algorithm efficiently converge on the global optima. The experimental results show that pcDE-ASL is both effective and efficient.
Real-time POI recommendation via modeling long- and short-term user preferences

Recently, Next Point-of-Interest (POI) Recommendation which proposes users for their next visiting locations, has gained increasing attention. A timely and accurate next POI recommendation can improve users’ efficient experiences. However, most existing methods typically focus on the sequential influence, but neglect the user’s real-time preference changing over time. In some scenarios, users may need a real-time POI recommendation, for example, when using Take-away Applications, users need recommending the appropriate restaurants at the specific moment. Hence, how to mine users’ patterns of life and their current preferences becomes an essential issue for the real-time POI recommendation. To address the issues above, we propose a real-time preference mining model (RTPM) which is based on LSTM to recommend the next POI with time restrictions. Specifically, RTPM mines users’ real-time preferences from long-term and short-term preferences in a uniform framework. For the long-term preferences, we mine the periodic trends of users’ behaviors between weeks to better reflect users’ patterns of life. While for the short-term preferences, trainable time transition vectors which represent the public preferences in corresponding time slots, are introduced to model users’ current time preferences influenced by the public. At the stage of recommendation, we design a category filter to filter out the POIs whose categories are unpopular in corresponding time slots to reduce the search space and make recommendation fit current time slot better. Note that RTPM does not utilize users’ attributes and their current locations for recommendation, which makes great contributions to users’ privacy protection. Extensive experiments on two real-world datasets demonstrate that RTPM outperforms the state-of-the-art models on Recall and NDCG.
Pyramid Spatial-Temporal Aggregation for Video-based Person Re-Identification

Video-based person re-identification aims to associate the video clips of the same person across multiple non-overlapping cameras. Spatial-temporal representations can provide richer and complementary information between frames, which are crucial to distinguish the target person when occlusion occurs. This paper proposes a novel Pyramid Spatial-Temporal Aggregation (PSTA) framework to aggregate the frame-level features progressively and fuse the hierarchical temporal features into a final video-level representation. Thus, short-term and long-term temporal information could be well exploited by different hierarchies. Furthermore, a Spatial-Temporal Aggregation Module (STAM) is proposed to enhance the aggregation capability of PSTA. It mainly consists of two novel attention blocks: Spatial Reference Attention (SRA) and Temporal Reference Attention (TRA). SRA explores the spatial correlations within a frame to determine the attention weight of each location. While TRA extends SRA with the correlations between adjacent frames, temporal consistency information can be fully explored to suppress the interference features and strengthen the discriminative ones. Extensive experiments on several challenging benchmarks demonstrate the effectiveness of the proposed PSTA, and our full model reaches 91.5% and 98.3% Rank-1 accuracy on MARS and DukeMTMC-VID benchmarks. The source code is available at https://github.com/WangYQ9/VideoReID-PSTA.
Naturalistic Physical Adversarial Patch for Object Detectors

Most prior works on physical adversarial attacks mainly focus on the attack performance but seldom enforce any restrictions over the appearance of the generated adversarial patches. This leads to conspicuous and attention-grabbing patterns for the generated patches which can be easily identified by humans. To address this issue, we pro-pose a method to craft physical adversarial patches for object detectors by leveraging the learned image manifold of a pretrained generative adversarial network (GAN) (e.g., BigGAN and StyleGAN) upon real-world images. Through sampling the optimal image from the GAN, our method can generate natural looking adversarial patches while maintaining high attack performance. With extensive experiments on both digital and physical domains and several independent subjective surveys, the results show that our proposed method produces significantly more realistic and natural looking patches than several state-of-the-art base-lines while achieving competitive attack performance. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
C2N: Practical Generative Noise Modeling for Real-World Denoising

Learning-based image denoising methods have been bounded to situations where well-aligned noisy and clean images are given, or samples are synthesized from predetermined noise models, e.g., Gaussian. While recent generative noise modeling methods aim to simulate the unknown distribution of real-world noise, several limitations still exist. In a practical scenario, a noise generator should learn to simulate the general and complex noise distribution without using paired noisy and clean images. However, since existing methods are constructed on the unrealistic assumption of real-world noise, they tend to generate implausible patterns and cannot express complicated noise maps. Therefore, we introduce a Clean-to-Noisy image generation framework, namely C2N, to imitate complex real-world noise without using any paired examples. We construct the noise generator in C2N accordingly with each component of real-world noise characteristics to express a wide range of noise accurately. Combined with our C2N, conventional denoising CNNs can be trained to outperform existing unsupervised methods on challenging real-world benchmarks by a large margin.
Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image

We present Worldsheet, a method for novel view synthesis using just a single RGB image as input. The main insight is that simply shrink-wrapping a planar mesh sheet onto the input image, consistent with the learned intermediate depth, captures underlying geometry sufficient to generate photorealistic unseen views with large viewpoint changes. To operationalize this, we propose a novel differentiable texture sampler that allows our wrapped mesh sheet to be textured and rendered differentiably into an image from a target viewpoint. Our approach is category-agnostic, end-to-end trainable without using any 3D supervision, and requires a single image at test time. We also explore a simple extension by stacking multiple layers of Worldsheets to better handle occlusions. Worldsheet consistently outperforms prior state-of-the-art methods on single-image view synthesis across several datasets. Furthermore, this simple idea captures novel views surprisingly well on a wide range of high-resolution in-the-wild images, converting them into navigable 3D pop-ups. Video results and code are available at https://worldsheet.github.io.
Student Customized Knowledge Distillation: Bridging the Gap Between Student and Teacher

Knowledge distillation (KD) transfers the dark knowledge from cumbersome networks (teacher) to lightweight (student) networks and expects the student to achieve more promising performance than training without the teacher’s knowledge. However, a counter-intuitive argument is that better teachers do not make better students due to the capacity mismatch. To this end, we present a novel adaptive knowledge distillation method to complement traditional approaches. The proposed method, named as Student Customized Knowledge Distillation (SCKD), examines the capacity mismatch between teacher and student from the perspective of gradient similarity. We formulate the knowledge distillation as a multi-task learning problem so that the teacher transfers knowledge to the student only if the student can benefit from learning such knowledge. We validate our methods on multiple datasets with various teacher-student configurations on image classification, object detection, and semantic segmentation.
GRACE: A Compressed Communication Framework for Distributed Machine Learning

Powerful computer clusters are used nowadays to train complex deep neural networks (DNN) on large datasets. Distributed training increasingly becomes communication bound. For this reason, many lossy compression techniques have been proposed to reduce the volume of transferred data. Unfortunately, it is difficult to argue about the behavior of compression methods, because existing work relies on inconsistent evaluation testbeds and largely ignores the performance impact of practical system configurations. In this paper, we present a comprehensive survey of the most influential compressed communication methods for DNN training, together with an intuitive classification (i.e., quantization, sparsification, hybrid and low-rank). Next, we propose GRACE, a unified framework and API that allows for consistent and easy implementation of compressed communication on popular machine learning toolkits. We instantiate GRACE on TensorFlow and PyTorch, and implement 16 such methods. Finally, we present a thorough quantitative evaluation with a variety of DNNs (convolutional and recurrent), datasets and system configurations. We show that the DNN architecture affects the relative performance among methods. Interestingly, depending on the underlying communication library and computational cost of compression / decompression, we demonstrate that some methods may be impractical. GRACE and the entire benchmarking suite are available as open-source.
Interventional Video Relation Detection

Video Visual Relation Detection (VidVRD) aims to semantically describe the dynamic interactions across visual concepts localized in a video in the form of subject, predicate, object. It can help to mitigate the semantic gap between vision and language in video understanding, thus receiving increasing attention in multimedia communities. Existing efforts primarily leverage the multimodal/spatio-temporal feature fusion to augment the representation of object trajectories as well as their interactions and formulate the prediction of predicates as a multi-class classification task. Despite their effectiveness, existing models ignore the severe long-tailed bias in VidVRD datasets. As a result, the models' prediction will be easily biased towards the popular head predicates (e.g., next-to and in-front-of), thus leading to poor generalizability.
Food recommendation with graph convolutional network

Food recommendation has attracted increasing attentions to various food-related applications and services. The food recommender models aim to match users’ preferences with recipes, where the key lies in the representation learning of users and recipes. However, ranging from early content-based filtering and collaborative filtering methods to recent hybrid methods, the existing work overlooks the various food-related relations, especially the ingredient-ingredient relations, leading to incomprehensive representations. To bridge this gap, we propose a novel model Food recommendation with Graph Convolutional Network (FGCN), which exploits ingredient-ingredient, ingredient-recipe, and recipe-user relations deeply. FGCN employs the information propagation mechanism and adopts multiple embedding propagation layers to model high-order connectivity across different food-related relations and enhance the representations. Specifically, we develop three types of information propagation: (1) ingredient-ingredient information propagation, (2) ingredient-recipe information propagation, and (3) recipe-user information propagation. To validate the effectiveness and rationality of FGCN, we conduct extensive experiments on a real-world dataset. The results show that the proposed FGCN outperforms the state-of-the-art baselines. Further in-depth analyses reveal that FGCN could alleviate the sparsity issue in food recommendation.
Sensitivity analysis: A discipline coming of age

Sensitivity analysis (SA) as a ‘formal’ and ‘standard’ component of scientific development and policy support is relatively young. Many researchers and practitioners from a wide range of disciplines have contributed to SA over the last three decades, and the SAMO (sensitivity analysis of model output) conferences, since 1995, have been the primary driver of breeding a community culture in this heterogeneous population. Now, SA is evolving into a mature and independent field of science, indeed a discipline with emerging applications extending well into new areas such as data science and machine learning. At this growth stage, the present editorial leads a special issue consisting of one Position Paper on “The future of sensitivity analysis” and 11 research papers on “Sensitivity analysis for environmental modelling” published in Environmental Modelling & Software in 2020–21.