Reviewerly/exHarmony huggingface dataset:
Indirect Adversarial Losses via an Intermediate Distribution for Training GANs

In this study, we consider the weak convergence characteristics of the Integral Probability Metrics (IPM) methods in training Generative Adversarial Networks (GANs). We first concentrate on a successful IPM-based GAN method that employs a repulsive version of the Maximum Mean Discrepancy (MMD) as the discriminator loss (called repulsive MMD-GAN). We reinterpret its repulsive metrics as an indirect discriminator loss function toward an intermediate distribution. This allows us to propose a novel generator loss via such an intermediate distribution based on our reinterpretation. Our indirect adversarial losses use a simple known distribution (i.e., the Normal or Uniform distribution in our experiments) to simulate indirect adversarial learning between three parts – real, fake, and intermediate distributions. Furthermore, we found the Kernelized Stein Discrepancy (KSD) from the IPM family as the adversarial loss function to avoid randomness from intermediate distribution samples because the target side (intermediate one) is sample-free in KSD. Experiments on several real-world datasets show that our methods can successfully train GANs with the intermediate-distribution-based KSD and MMD and can outperform previous loss metrics.
Dense but Efficient VideoQA for Intricate Compositional Reasoning

It is well known that most of the conventional video question answering (VideoQA) datasets consist of easy questions requiring simple reasoning processes. However, long videos inevitably contain complex and compositional semantic structures along with the spatio-temporal axis, which requires a model to understand the compositional structures inherent in the videos. In this paper, we suggest a new compositional VideoQA method based on transformer architecture with a deformable attention mechanism to address the complex VideoQA tasks. The deformable attentions are introduced to sample a subset of informative visual features from the dense visual feature map to cover a temporally long range of frames efficiently. Furthermore, the dependency structure within the complex question sentences is also combined with the language embeddings to readily understand the relations among question words. Extensive experiments and ablation studies show that the suggested dense but efficient model outperforms other baselines.
IDD-3D: Indian Driving Dataset for 3D Unstructured Road Scenes

Autonomous driving and assistance systems rely on annotated data from traffic and road scenarios to model and learn the various object relations in complex real-world scenarios. Preparation and training of deploy-able deep learning architectures require the models to be suited to different traffic scenarios and adapt to different situations. Currently, existing datasets, while large-scale, lack such diversities and are geographically biased towards mainly developed cities. An unstructured and complex driving layout found in several developing countries such as India poses a challenge to these models due to the sheer degree of variations in the object types, densities, and locations. To facilitate better research toward accommodating such scenarios, we build a new dataset, IDD-3D, which consists of multimodal data from multiple cameras and LiDAR sensors with 12k annotated driving LiDAR frames across various traffic scenarios. We discuss the need for this dataset through statistical comparisons with existing datasets and highlight benchmarks on standard 3D object detection and tracking tasks in complex layouts. Code and data available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Fine-grained Activities of People Worldwide

Every day, humans perform many closely related activities that involve subtle discriminative motions, such as putting on a shirt vs. putting on a jacket, or shaking hands vs. giving a high five. Activity recognition by ethical visual AI could provide insights into our patterns of daily life, however existing activity recognition datasets do not capture the massive diversity of these human activities around the world. To address this limitation, we introduce Collector, a free mobile app to record video while simultaneously annotating objects and activities of consented subjects. This new data collection platform was used to curate the Consented Activities of People (CAP) dataset, the first large-scale, fine-grained activity dataset of people worldwide. The CAP dataset contains 1.45M video clips of 512 fine grained activity labels of daily life, collected by 780 subjects in 33 countries. We provide activity classification and activity detection benchmarks for this dataset, and analyze baseline results to gain insight into how people around with world perform common activities. The dataset, benchmarks, evaluation tools, public leaderboards and mobile apps are available for use at https://visym.github.io/cap.
Learning Few-shot Segmentation from Bounding Box Annotations

We present a new weakly-supervised few-shot semantic segmentation setting and a meta-learning method for tackling the new challenge. Different from existing settings, we leverage bounding box annotations as weak supervision signals during the meta-training phase, i.e., more label-efficient. Bounding box provides a cheaper label representation than segmentation mask but contains both an object of interest and a disturbing background. We first show that meta-training with bounding boxes degrades recent few-shot semantic segmentation methods, which are typically meta-trained with full semantic segmentation supervisions. We postulate that this challenge is originated from the impure information of bounding box representation. We propose a pseudo trimap estimator and trimap-attention based prototype learning to extract clearer supervision signals from bounding boxes. These developments robustify and generalize our method well to noisy support masks at test time. We empirically show that our method consistently improves performance. Our method gains 1.4% and 3.6% mean-IoU over the competing one in full and weak test supervision cases, respectively, in the 1-way 5-shot setting on Pascal-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> .
No Reference Opinion Unaware Quality Assessment of Authentically Distorted Images

The quality assessment (QA) of camera captured authentically distorted images is important on account of its ubiquitous applications and challenging due to the lack of a reference. While there exists a plethora of supervised no reference (NR) image QA (IQA) algorithms, there is a need to study unsupervised or opinion unaware algorithms on account of their superior generalization performance. We explore self-supervised learning (SSL) for the feature design on authentically distorted images to predict quality without training on human labels. While SSL on synthetic distortions has recently shown promise, there is a need to enrich the feature learning on authentic distortions. The key challenge in achieving this is in the learning of quality sensitive features with mitigated content dependence. We design a self-supervised contrastive learning approach which only requires positives and introduce a content separation loss by estimating a bound on the mutual information between the features learnt and the content information. We show on multiple authentically distorted datasets that our self-supervised features can predict image quality by comparing with a corpus of pristine images and achieve state-of-the-art performance. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">§</sup>
Split to Learn: Gradient Split for Multi-Task Human Image Analysis

This paper presents an approach to train a unified deep network that simultaneously solves multiple human-related tasks. A multi-task framework is favorable for sharing information across tasks under restricted computational resources. However, tasks not only share information but may also compete for resources and conflict with each other, making the optimization of shared parameters difficult and leading to suboptimal performance. We propose a simple but effective training scheme called GradSplit that alleviates this issue by utilizing asymmetric inter-task relations. Specifically, at each convolution module, it splits features into T groups for T tasks and trains each group only using the gradient back-propagated from the task losses with which it does not have conflicts. During training, we apply GradSplit to a series of convolution modules. As a result, each module is trained to generate a set of task-specific features using the shared features from the previous module. This enables a network to use complementary information across tasks while circumventing gradient conflicts. Experimental results show that GradSplit achieves a better accuracy-efficiency trade-off than existing methods. It minimizes accuracy drop caused by task conflicts while significantly saving compute resources in terms of both FLOPs and memory at inference. We further show that GradSplit achieves higher cross-dataset accuracy compared to single-task and other multi-task networks.
Frequency-Aware Self-Supervised Monocular Depth Estimation

We present two versatile methods to generally enhance self-supervised monocular depth estimation (MDE) models. The high generalizability of our methods is achieved by solving the fundamental and ubiquitous problems in photometric loss function. In particular, from the perspective of spatial frequency, we first propose Ambiguity-Masking to suppress the incorrect supervision under photometric loss at specific object boundaries, the cause of which could be traced to pixel-level ambiguity. Second, we present a novel frequency-adaptive Gaussian low-pass filter, designed to robustify the photometric loss in high-frequency regions. We are the first to propose blurring images to improve depth estimators with an interpretable analysis. Both modules are lightweight, adding no parameters and no need to manually change the network structures. Experiments show that our methods provide performance boosts to a large number of existing models, including those who claimed state-of-the-art, while introducing no extra inference computation at all.
A Novel Capability of Object Identification and Recognition Based on Integrated mWMM

In the last decade, there has been remarkable progress in the areas of object detection and recognition due to high-quality color images along with their depth maps provided by RGB-D cameras.They enable artificially intelligent machines to easily detect and recognize objects and make real-time decisions according to the given scenarios.Depth cues can improve the quality of object detection and recognition.The main purpose of this research study to find an optimized way of object detection and identification we propose techniques of object detection using two RGB-D datasets.The proposed methodology extracts image normally from depth maps and then performs clustering using the Modified Watson Mixture Model (mWMM).mWMM is challenging to handle when the quality of the image is not good.Hence, the proposed RGB-D-based system uses depth cues for segmentation with the help of mWMM.Then it extracts multiple features from the segmented images.The selected features are fed to the Artificial Neural Network (ANN) and Convolutional Neural Network (CNN) for detecting objects.We achieved 92.13% of mean accuracy over NYUv1 dataset and 90.00% of mean accuracy for the Redweb_v1 dataset.Finally, their results are compared and the proposed model with CNN outperforms other state-of-the-art methods.The proposed architecture can be used in autonomous cars, traffic monitoring, and sports scenes.
Home Automation-Based Health Assessment Along Gesture Recognition via Inertial Sensors

Hand gesture recognition (HGR) is used in a numerous applications, including medical health-care, industrial purpose and sports detection.We have developed a real-time hand gesture recognition system using inertial sensors for the smart home application.Developing such a model facilitates the medical health field (elders or disabled ones).Home automation has also been proven to be a tremendous benefit for the elderly and disabled.Residents are admitted to smart homes for comfort, luxury, improved quality of life, and protection against intrusion and burglars.This paper proposes a novel system that uses principal component analysis, linear discrimination analysis feature extraction, and random forest as a classifier to improve HGR accuracy.We have achieved an accuracy of 94% over the publicly benchmarked HGR dataset.The proposed system can be used to detect hand gestures in the healthcare industry as well as in the industrial and educational sectors.
Enhanced E-commerce Fraud Prediction Based on a Convolutional Neural Network Model

The rapidly escalating sophistication of e-commerce fraud in recent years has led to an increasing reliance on fraud detection methods based on machine learning.However, fraud detection methods based on conventional machine learning approaches suffer from several problems, including an excessively high number of network parameters, which decreases the efficiency and increases the difficulty of training the network, while simultaneously leading to network overfitting.In addition, the sparsity of positive fraud incidents relative to the overwhelming proportion of negative incidents leads to detection failures in trained networks.The present work addresses these issues by proposing a convolutional neural network (CNN) framework for detecting ecommerce fraud, where network training is conducted using historical market transaction data.The number of network parameters reduces via the local perception field and weight sharing inherent in the CNN framework.In addition, this deep learning framework enables the use of an algorithmiclevel approach to address dataset imbalance by focusing the CNN model on minority data classes.The proposed CNN model is trained and tested using a large public e-commerce service dataset from 2018, and the test results demonstrate that the model provides higher fraud prediction accuracy than existing state-of-the-art methods.
Reconstructing Humpty Dumpty: Multi-feature Graph Autoencoder for Open Set Action Recognition

Most action recognition datasets and algorithms assume a closed world, where all test samples are instances of the known classes. In open set problems, test samples may be drawn from either known or unknown classes. Existing open set action recognition methods are typically based on extending closed set methods by adding post hoc analysis of classification scores or feature distances and do not capture the relations among all the video clip elements. Our approach uses the reconstruction error to determine the novelty of the video since unknown classes are harder to put back together and thus have a higher reconstruction error than videos from known classes. We refer to our solution to the open set action recognition problem as "Humpty Dumpty", due to its reconstruction abilities. Humpty Dumpty is a novel graph-based autoencoder that accounts for contextual and semantic relations among the clip pieces for improved reconstruction. A larger reconstruction error leads to an increased likelihood that the action can not be reconstructed, i.e., can not put Humpty Dumpty back together again, indicating that the action has never been seen before and is novel/unknown. Extensive experiments are performed on two publicly available action recognition datasets including HMDB-51 and UCF-101, showing the state-of-the-art performance for open set action recognition.
Multi-scale Cell-based Layout Representation for Document Understanding

Deep learning techniques have achieved remarkable progress in document understanding. Most models use co-ordinates to represent absolute or relative spatial information of components, but they are difficult to represent latent rules in the document layout. This makes learning layout representation to be more difficult. Unlike the previous researches which have employed the coordinate system, graph or grid to represent the document layout, we propose a novel layout representation, the cell-based layout, to provide easy-to-understand spatial information for backbone models. In line with human reading habits, it uses cell information, i.e. row and column index, to represent the position of components in a document, and makes the document layout easier to understand. Furthermore, we proposed the multi-scale layout to represent the hierarchical structure of layout, and developed a data augmentation method to improve the performance. Experiment results show that our method achieves the state-of-the-art performance in text-based tasks, including form understanding and receipt understanding, and improves the performance in image-based task such as document image classification. We released the code in the repo <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</sup> .
A Dual Model Watermarking Framework for Copyright Protection in Image Processing Networks

Image processing networks have gained great success in many fields, and thus the issue of copyright protection for image processing networks has become a focus of attention.Model watermarking techniques are widely used in model copyright protection, but there are two challenges: (1) designing universal trigger sample watermarking for different network models is still a challenge; (2) existing methods of copyright protection based on trigger s watermarking are difficult to resist forgery attacks.In this work, we propose a dual model watermarking framework for copyright protection in image processing networks.The trigger sample watermark is embedded in the training process of the model, which can effectively verify the model copyright.And we design a common method for generating trigger sample watermarks based on generative adversarial networks, adaptively generating trigger sample watermarks according to different models.The spatial watermark is embedded into the model output.When an attacker steals model copyright using a forged trigger sample watermark, which can be correctly extracted to distinguish between the piratical and the protected model.The experiments show that the proposed framework has good performance in different image segmentation networks of UNET, UNET++, and FCN (fully convolutional network), and effectively resists forgery attacks.
Semantic Segmentation by Using Down-Sampling and Subpixel Convolution: DSSC-UNet

Recently, semantic segmentation has been widely applied to image processing, scene understanding, and many others.Especially, in deep learning-based semantic segmentation, the U-Net with convolutional encoder-decoder architecture is a representative model which is proposed for image segmentation in the biomedical field.It used max pooling operation for reducing the size of image and making noise robust.However, instead of reducing the complexity of the model, max pooling has the disadvantage of omitting some information about the image in reducing it.So, this paper used two diagonal elements of down-sampling operation instead of it.We think that the down-sampling feature maps have more information intrinsically than max pooling feature maps because of keeping the Nyquist theorem and extracting the latent information from them.In addition, this paper used two other diagonal elements for the skip connection.In decoding, we used Subpixel Convolution rather than transposed convolution to efficiently decode the encoded feature maps.Including all the ideas, this paper proposed the new encoder-decoder model called Down-Sampling and Subpixel Convolution U-Net (DSSC-UNet).To prove the better performance of the proposed model, this paper measured the performance of the U-Net and DSSC-UNet on the Cityscapes.As a result, DSSC-UNet achieved 89.6% Mean Intersection Over Union (Mean-IoU) and U-Net achieved 85.6% Mean-IoU, confirming that DSSC-UNet achieved better performance.
Detecting Double JPEG Compressed Color Images via an Improved Approach

Detecting double Joint Photographic Experts Group (JPEG) compression for color images is vital in the field of image forensics.In previous researches, there have been various approaches to detecting double JPEG compression with different quantization matrices.However, the detection of double JPEG color images with the same quantization matrix is still a challenging task.An effective detection approach to extract features is proposed in this paper by combining traditional analysis with Convolutional Neural Networks (CNN).On the one hand, the number of nonzero pixels and the sum of pixel values of color space conversion error are provided with 12dimensional features through experiments.On the other hand, the rounding error, the truncation error and the quantization coefficient matrix are used to generate a total of 128-dimensional features via a specially designed CNN.In such a CNN, convolutional layers with fixed kernel of 1×1 and Dropout layers are adopted to prevent overfitting of the model, and an average pooling layer is used to extract local characteristics.In this approach, the Support Vector Machine (SVM) classifier is applied to distinguish whether a given color image is primarily or secondarily compressed.The approach is also suitable for the case when customized needs are considered.The experimental results show that the proposed approach is more effective than some existing ones when the compression quality factors are low.
Federated Learning Based on Data Divergence and Differential Privacy in燜inancial Risk Control Research

In the financial sector, data are highly confidential and sensitive, and ensuring data privacy is critical.Sample fusion is the basis of horizontal federation learning, but it is suitable only for scenarios where customers have the same format but different targets, namely for scenarios with strong feature overlapping and weak user overlapping.To solve this limitation, this paper proposes a federated learning-based model with local data sharing and differential privacy.The indexing mechanism of differential privacy is used to obtain different degrees of privacy budgets, which are applied to the gradient according to the contribution degree to ensure privacy without affecting accuracy.In addition, data sharing is performed to improve the utility of the global model.Further, the distributed prediction model is used to predict customers' loan propensity on the premise of protecting user privacy.Using an aggregation mechanism based on federated learning can help to train the model on distributed data without exposing local data.The proposed method is verified by experiments, and experimental results show that for non-iid data, the proposed method can effectively improve data accuracy and reduce the impact of sample tilt.The proposed method can be extended to edge computing, blockchain, and the Industrial Internet of Things (IIoT) fields.The theoretical analysis and experimental results show that the proposed method can ensure the privacy and accuracy of the federated learning process and can also improve the model utility for non-iid data by 7% compared to the federated averaging method (FedAvg).