Reviewerly/exHarmony huggingface dataset:
Active Spatial Positions Based Hierarchical Relation Inference for Group Activity Recognition

Group activity recognition aims to recognize behaviors characterized by multiple individuals within a scene. Existing schemes rely on individual relation inference and usually take the individuals as tokens. Essentially they select the most relevant region of the group activity from the entire image while filtering out irrelevant background noises. However, these schemes require individual bounding box labeling in both training and testing stages. Since individuals have usually been presented at one scale, multi-scale individuals cannot be combined in an effective way. In this paper, we present a novel end-to-end hierarchical relation inference framework based on active spatial positions for group activity recognition. This framework is designed to locate active spatial positions and use them as visual tokens to infer the relations for token embeddings. It requires individual bounding box labeling only in the training stage while automatically eliminating the background after locating active spatial positions from the entire scene. The hierarchical relations can be naturally inferred based on the visual tokens at different scales, contributing to further performance improvement. Experimental results demonstrate that the proposed framework is competitive against existing schemes that require more laboring and computation to generate labels in both the training and testing stage.
FedBroadcast: Exploit Broadcast Channel for Fast Convergence in Wireless Federated Learning

With the fast development of modern networking technologies, the transmission rate and reliability of wireless networks have been greatly improved. Meanwhile, the fast-developing Internet of Things (IoT) devices provide continually increasing computation capability. Federated learning (FL) was proposed to leverage communication and computation resources to perform machine learning (ML) tasks on IoT devices. In wireless FL systems, devices train ML models with local data sets, and a base station (BS) aggregates these trained models so that data privacy is guaranteed by the isolation of data sets. However, in the existing works, the same global model is transmitted to devices individually over the wireless channel multiple times, while the updated local models are received only by the BS, which ignores the wireless broadcast channel, incurs large communication overhead, and slows down the overall training process. In this article, we propose the FedBroadcast protocol to efficiently exploit the shared wireless broadcast channel for the two-way model transmission in FL. In the downloading step, we let BS broadcast the global model once for all scheduled devices, and design a dynamic programming-based algorithm to schedule devices optimally. In the uploading step, we also leverage the wireless broadcast channel and select some devices to receive all the updated local models without waiting for model downloading in the next round. Finally, to solve the potential block-cyclic sampling problem brought by device scheduling, we adopt pluralistic averaging modification, which improves the convergence performance under extreme data distributions. Extensive experiments demonstrate that FedBroadcast outperforms the existing wireless FL under different system settings.
SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis

We present a dataset of 1000 video sequences of human portraits recorded in real and uncontrolled conditions by using a handheld smartphone accompanied by an external high-quality depth camera. The collected dataset contains 200 people captured in different poses and locations and its main purpose is to bridge the gap between raw measurements obtained from a smartphone and downstream applications, such as state estimation, 3D reconstruction, view synthesis, etc. The sensors employed in data collection are the smartphone's camera and Inertial Measurement Unit (IMU), and an external Azure Kinect DK depth camera software synchronized with sub-millisecond precision to the smartphone system. During the recording, the smartphone flash is used to provide a periodic secondary source of lightning. Accurate mask of the foremost person is provided as well as its impact on the camera alignment accuracy. For evaluation purposes, we compare multiple state-of-the-art camera alignment methods by using a Motion Cap-ture system. We provide a smartphone visual-inertial bench-mark for portrait capturing, where we report results for multiple methods and motivate further use of the provided trajectories, available in the dataset, in view synthesis and 3D reconstruction tasks.
CycleGAN Clinical Image Augmentation Based on Mask Self-Attention Mechanism

With the development of society and the advancement of science and technology, artificial intelligence has also emerged as the times require. In computer vision, deep learning based on convolutional neural networks(CNN) achieves state-of-the-art performance. However, the massive data requirements of deep learning have long been a pain point in the field, especially in the medical field, where it is often difficult (and sometimes impossible) to obtain enough training data for some specific tasks. To overcome insufficient and unbalanced data, in this paper, we focus on the generation and balance of data on radiation-induced pneumonia, an extremely rare disease with a low incidence. As a result, datasets on this disease are extremely sparse and unevenly distributed. To address the above problems, the predecessors’ method is often to use generative models to generate data as a complement of the fewer samples to achieve a balanced distribution of data samples. Among various generative models, CycleGAN is widely used in medical image generation due to its cycle consistency to achieve style migration without changing the basic content. However, the original CycleGAN method has many shortcomings, especially in Few-shot and the data unevenly distributed, its performance will be greatly reduced. To make the generated data samples retain the original structure and have finer and clearer details, this paper proposes a mask-based self-attention CycleGAN data augmentation method. A self-attention branch is added to the generator and two different loss functions named Self-Attention Loss and Mask Loss are designed. To stabilize the training process, spectral normalization is introduced to improve the discriminator and MS-SSIM and L1 joint loss are used to improve the original identity loss. The ResNet18 is used to complete classification experiments on the radiation-induced pneumonia dataset and the COVID-19 dataset respectively. Four classification performance indicators: the area under the ROC curve (AUC), Accuracy (ACC), Sensitivity (SEN), and Specificity (SPE) are calculated to verify the effectiveness and generalization of our method. Compared with the original CycleGAN and traditional data augmentation, the classifier trained by data augmentation using our method has outstanding performance in multiple classification indicators and has better classification performance. Experimental results show that our method solves the problem of insufficient samples and data imbalance in the pneumonia dataset by generating high-quality pneumonia images. Code is available at https://github.com/ngfufdrdh/CycleGAN-lung.
Novel RNS Reverse Conversion Algorithm Based on Core Function

Residue Number System (RNS) provides advantages for accelerating arithmetic computations using multiplications and additions only due to absence of long carry propagation logic in its modular adders and multipliers. However, a conversion from RNS to Binary Number System (BNS) is a well-known complex operation requiring larger hardware area and consuming more power than modular multipliers which significantly limits applicability of RNS. Many non-modular operation (e.g. comparison, division) implementations are also based on reverse conversion. This article presents a novel algorithm for RNS to BNS reverse conversion for an arbitrary modular set based on a Core function. We proposed core function coefficients selection method, which allows configuration of the Core function dynamic range and simplifies the conversion complexity. New algorithm allows to achieve up to 13% of hardware area and up to 51% of power consumption reduction and become up to 17% faster than existing best optimized implementations of RNS reverse convertors for the large arbitrary moduli set.
Ship object detection in one-stage framework based on Swin-Transformer

In the field of maritime computer vision, practicality is significant. Both shore-based surveillance systems and autonomous ships require highly accurate recognition and rapid detection. Swin-Transformer is a superior neural network in computer vision lately, and demonstrates powerful feature extraction capabilities. It has become a new option for the tasks of detection. On the basis of our previous proven results in applying Swin-Transformer for ship object detection in a two-stage framework, the purpose of this paper is more practical. So, we incorporate Swin-Transformer into one-stage frameworks, enabling real-time detection of ship targets in the maritime environment. By using the famous Seaships dataset for training, Swin-Transformer outperforms CNN-based models with mean average precision of 92.37% and 80.59%, serving as the backbone for YOLOv3 and SSD frameworks, respectively. Notably, detection speed can reach 23 frames per second (fps) and 27 fps, much faster than 9 fps in two-stage framework. The foundation for developing Swin-Transformer in ship object detection has been laid.
A Precise Calibration Method for the Underwater Binocular Camera With Unknown Refractive Index

Underwater camera calibration usually needs to solve an accurate refraction model to improve the calibration accuracy. In the refraction model, there are some key parameters: 1) the distance from the optical center to the refraction plane; 2) the thickness of the housing; 3) the normal vector of the refraction plane; and 4) the refractive index of the medium. In many cases, the refractive index is often assumed to be a known parameter. However, the refractive index of water will change due to environmental changes such as temperature, salinity, or depth. Inaccurate refractive index will seriously affect the measurement accuracy. Therefore, a precise calibration method for the underwater binocular camera with unknown refractive index is proposed. All the unknown parameters of the refraction model, including the refractive index of the medium, are solved by the nondominated sorting genetic algorithm (NSGA)-III. The parameter solving process is regarded as a multiobjective optimization problem, and the optimization objectives are determined by using the relative position and the actual position relationship of the checkerboard corners. Experiments show that the proposed method can greatly improve the 3-D measurement accuracy of binocular camera in unknown waters.
JUICE: JUstIfied Counterfactual Explanations

Complex, highly accurate machine learning algorithms support decision-making processes with large and intricate datasets. However, these models have low explainability. Counterfactual explanation is a technique that tries to find a set of feature changes on a given instance to modify the models prediction output from an undesired to a desired class. To obtain better explanations, it is crucial to generate faithful counterfactuals, supported by and connected to observations and the knowledge constructed on them. In this study, we propose a novel counterfactual generation algorithm that provides faithfulness by justification, which may increase developers and users trust in the explanations by supporting the counterfactuals with a known observation. The proposed algorithm guarantees justification for mixed-features spaces and we show it performs similarly with respect to state-of-the-art algorithms across other metrics such as proximity, sparsity, and feasibility. Finally, we introduce the first model-agnostic algorithm to verify counterfactual justification in mixed-features spaces.
Free-Viewpoint RGB-D Human Performance Capture and Rendering

Capturing and faithfully rendering photorealistic humans from novel views is a fundamental problem for AR/VR applications. While prior work has shown impressive performance capture results in laboratory settings, it is non-trivial to achieve casual free-viewpoint human capture and rendering for unseen identities with high fidelity, especially for facial expressions, hands, and clothes. To tackle these challenges we introduce a novel view synthesis framework that generates realistic renders from unseen views of any human captured from a single-view and sparse RGB-D sensor, similar to a low-cost depth camera, and without actor-specific models. We propose an architecture to create dense feature maps in novel views obtained by sphere-based neural rendering, and create complete renders using a global context inpainting model. Additionally, an enhancer network leverages the overall fidelity, even in occluded areas from the original view, producing crisp renders with fine details. We show that our method generates high-quality novel views of synthetic and real human actors given a single-stream, sparse RGB-D input. It generalizes to unseen identities, and new poses and faithfully reconstructs facial expressions. Our approach outperforms prior view synthesis methods and is robust to different levels of depth sparsity.
Are Deep Models Robust against Real Distortions? A Case Study on Document Image Classification

As deep learning models in the context of document image classification are reaching diminishing returns with nearperfect recognition scores, their robustness characteristics are poorly understood. In order to evaluate the robustness of existing state-of-the-art document image classifiers against different types of distortions that are commonly encountered in the real world, we present two separate benchmark datasets, namely RVL-CDIPD and Tobacco3482-D. The proposed benchmarks are generated by augmenting the well-known pre-existing document image classification datasets (RVL-CDIP and Tobacco3482) with 21 different types of distortions including varying severity levels. We leverage the proposed benchmark datasets to analyze the robustness characteristics of existing document image classification systems. Our analysis reveals that despite higher accuracy models exhibiting relatively higher robustness, they still severely underperform on some specific distortions, with classification accuracies dropping from ~90% to as low as ~40% in some cases. Interestingly, some of these high accuracy models perform even worse than the baseline AlexNet model in the presence of distortions, with the relative decline in their accuracy sometimes reaching as high as 300-450%. We envision these benchmarks to serve as a strong signal of progress in document image classification tasks, beyond the saturated accuracy metrics. The datasets and code to reproduce them is publicly available: https://github.com/saifullah3396/docrobustness.
Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly

Machine learning has advanced dramatically, narrowing the accuracy gap to humans in multimodal tasks like visual question answering (VQA). However, while humans can say "I don't know" when they are uncertain (i.e., abstain from answering a question), such ability has been largely neglected in multimodal research, despite the importance of this problem to the usage of VQA in real settings. In this work, we promote a problem formulation for reliable VQA, where we prefer abstention over providing an incorrect answer. We first enable abstention capabilities for several VQA models, and analyze both their coverage, the portion of questions answered, and risk, the error on that portion. For that, we explore several abstention approaches. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain by directly using a model's softmax scores limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%). This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can increase the coverage by, for example, $$2.4\times $$ from 6.8% to 16.3% at 1% risk. While it is important to analyze both coverage and risk, these metrics have a trade-off which makes comparing VQA models challenging. To address this, we also propose an Effective Reliability metric for VQA that places a larger cost on incorrect answers compared to abstentions. This new problem formulation, metric, and analysis for VQA provide the groundwork for building effective and reliable VQA models that have the self-awareness to abstain if and only if they don't know the answer. Code and Models: https://github.com/facebookresearch/reliable_vqa .
Adversarial Partial Domain Adaptation by Cycle Inconsistency

Unsupervised partial domain adaptation (PDA) is a unsupervised domain adaptation problem which assumes that the source label space subsumes the target label space. A critical challenge of PDA is the negative transfer problem, which is triggered by learning to match the whole source and target domains. To mitigate negative transfer, we note a fact that, it is impossible for a source sample of outlier classes to find a target sample of the same category due to the absence of outlier classes in the target domain, while it is possible for a source sample of shared classes. Inspired by this fact, we exploit the cycle inconsistency, i.e., category discrepancy between the original features and features after cycle transformations, to distinguish outlier classes apart from shared classes in the source domain. Accordingly, we propose to filter out source samples of outlier classes by weight suppression and align the distributions of shared classes between the source and target domains by adversarial learning. To learn accurate weight assignment for filtering out outlier classes, we design cycle transformations based on domain prototypes and soft nearest neighbor, where center losses are introduced in individual domains to reduce the intra-class variation. Experiment results on three benchmark datasets demonstrate the effectiveness of our proposed method.
An Invariant Matching Property for Distribution Generalization under Intervened Response

The task of distribution generalization concerns making reliable prediction of a response in unseen environments. The structural causal models are shown to be useful to model distribution changes through intervention. Motivated by the fundamental invariance principle, it is often assumed that the conditional distribution of the response given its predictors remains the same across environments. However, this assumption might be violated in practical settings when the response is intervened. In this work, we investigate a class of model with an intervened response. We identify a novel form of invariance by incorporating the estimates of certain features as additional predictors. Effectively, we show this invariance is equivalent to having a deterministic linear matching that makes the generalization possible. We provide an explicit characterization of the linear matching and present our simulation results under various intervention settings.
A Real-Time Oil Content Analysis Method of Cuttings Based on Deep Learning

In the oil exploration site, logging technology is usually used to obtain geological data, and the obtained cuttings are analyzed by manual identification. Due to the different experience of logging personnel and their different cognition of the results, only qualitative analysis can be conducted. In this paper, a real-time analysis method of oil content in cuttings based on deep learning is proposed. Firstly, the cuttings image is collected under the fluorescent lamp, and then the semantic segmentation network MobileNetV3 UNet (M3-UNet) is used to segment the oil-bearing region of cuttings image automatically. Finally, the real-time quantitative analysis of oil content in cuttings is realized. In order to make the proposed algorithm be applied to the oil exploration site, the proposed algorithm is deployed on the Jetson edge calculation equipment to realize the portable oil content in cuttings analysis equipment. In the coding stage of the proposed M3-UNet, MobileNetV3 is used as the backbone network to extract the features from the input cuttings image, which can reduce the requirements on the performance of logging equipment while maintaining the segmentation accuracy. The experimental results show that the Intersection over Union (IoU), F1-score and Pixel Accuracy (PA) of M3-UNet segmentation in the test dataset are all more than 9% higher than U-Net and MobileNetV2 UNet (M2-UNet). The segmentation speed of the cuttings image on Jetson equipment reaches 21FPS, meeting the requirements of real-time analysis of oil content of the cuttings image on the logging site.
It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection

Datasets that capture the connection between vision, language, and affection are limited, causing a lack of understanding of the emotional aspect of human intelligence. As a step in this direction, the ArtEmis dataset was recently introduced as a large-scale dataset of emotional reactions to images along with language explanations of these chosen emotions. We observed a significant emotional bias towards instance-rich emotions, making trained neural speakers less accurate in describing under-represented emotions. We show that collecting new data, in the same way, is not effective in mitigating this emotional bias. To remedy this problem, we propose a contrastive data collection approach to balance ArtEmis with a new complementary dataset such that a pair of similar images have contrasting emotions (one positive and one negative). We collected 260,533 instances using the proposed method, we combine them with ArtEmis, creating a second iteration of the dataset. The new combined dataset, dubbed ArtEmis v2.0, has a balanced distribution of emotions with explanations revealing more fine details in the associated painting. Our experiments show that neural speakers trained on the new dataset improve CIDEr and METEOR evaluation metrics by 20% and 7%, respectively, compared to the biased dataset. Finally, we also show that the performance per emotion of neural speakers is improved across all the emotion categories, significantly on under-represented emotions. The collected dataset and code are available at https://artemisdataset-v2.org.
RamGAN: Region Attentive Morphing GAN for Region-Level Makeup Transfer

In this paper, we propose a region adaptive makeup transfer GAN, called RamGAN, for precise region-level makeup transfer. Compared to face-level transfer methods, our RamGAN uses spatial-aware Region Attentive Morphing Module (RAMM) to encode Region Attentive Matrices (RAMs) for local regions like lips, eye shadow and skin. After that, the Region Style Injection Module (RSIM) is applied to RAMs produced by RAMM to obtain two Region Makeup Tensors, $$\gamma $$ and $$\beta $$ , which are subsequently added to the feature map of source image to transfer the makeup. As attention and makeup styles are calculated for each region, RamGAN can achieve better disentangled makeup transfer for different facial regions. When there are significant pose and expression variations between source and reference, RamGAN can also achieve better transfer results, due to the integration of spatial information and region-level correspondence. Experimental results are conducted on public datasets like MT, M-Wild and Makeup datasets, both visual and quantitative results and user study suggest that our approach achieves better transfer results than state-of-the-art methods like BeautyGAN, BeautyGlow, DMT, CPM and PSGAN.