Reviewerly/exHarmony huggingface dataset:
The Topology and Language of Relationships in the Visual Genome Dataset

The Visual Genome Dataset is the de facto standard dataset used in Scene Graph generation. It contains a large collection of images with corresponding object and relationship labels. We explore the lingual aspect of the relationship predicates and find that very few symmetric/inverse relationships are represented in the dataset(for example, ’above’ and ’under’). We believe this is linked to human spatial cognition, and posit that labelling bias stemming from human representations of relationships creates asymmetric relationship labels that span the whole dataset. We also perform a 2D topological analysis of the bounding boxes linked by different relationship predicates. This analysis sheds light on certain classes and their ambiguity wherein more frequent classes are semantically overloaded and therefore quite confusing. Finally we show that when reduced to more lingually and topologically well defined spatial relationships scene graph generation algorithm performance improves tremendously, but scene graph generators are still far from perfect.
Exploring Motion Information for Distractor Suppression in Visual Tracking

In the past few years, Siamese networks have achieved outstanding improvements in visual object tracking. However, visual distractors with similar semantics can be easily misclassified as the target by Siamese networks and may consequently result in the drift problem. Besides, the Hanning window penalty, which is generally used to suppress distractors, could fail in many challengeable scenes. Notably, most failures violate the assumption of motion continuity. Thus, in this work, we explore motion information to mitigate the drift problem in visual tracking. First, we introduce a simple linear Kalman filter to predict the bounding box of the target in the current frame, which acts as a reference for decisions. Second, an IoU-Guided penalty is assembled in the post-processing to suppress distractors effectively. It’s worth mentioning that our method is almost cost-free. We conduct numerous experimental validations and analyses of our approach on several challenging sequences and datasets. Our tracker runs at approximately 40 fps and performs well on those sequences which include the Background Clutter attribute. Finally, by simultaneously integrating the IoU-Guidedpenalty and the Hanning window penalty with a strong baseline tracker TransT [7], our method achieves favorable gains by 69.1→71.5, 65.7→66.7, 64.9→65.9 success on OTB-100 [32], LaSOT [12], NFS[18].
medXGAN: Visual Explanations for Medical Classifiers through a Generative Latent Space

Despite the surge of deep learning in the past decade, some users are skeptical to deploy these models in practice due to their black-box nature. Specifically, in the medical space where there are severe potential repercussions, we need to develop methods to gain confidence in the models&#x2019; decisions. To this end, we propose a novel medical imaging generative adversarial framework, medXGAN (medical eXplanation GAN), to visually explain what a medical classifier focuses on in its binary predictions. By encoding domain knowledge of medical images, we are able to disentangle anatomical structure and pathology, leading to fine-grained visualization through latent interpolation. Furthermore, we optimize the latent space such that interpolation explains how the features contribute to the classifier&#x2019;s output. Our method outperforms baselines such as Gradient-Weighted Class Activation Mapping (Grad-CAM) and Integrated Gradients in localization and explanatory ability. Additionally, a combination of the medXGAN with Integrated Gradients can yield explanations more robust to noise. The project page with code is available at: https://avdravid.github.io/medXGANpage/.
Discriminability-enforcing loss to improve representation learning

During the training process, deep neural networks implicitly learn to represent the input data samples through a hierarchy of features, where the size of the hierarchy is determined by the number of layers. In this paper, we focus on enforcing the discriminative power of the high-level representations, that are typically learned by the deeper layers (closer to the output). To this end, we introduce a new loss term inspired by the Gini impurity, which is aimed at minimizing the entropy (increasing the discriminative power) of individual high-level features with respect to the class labels. Although our Gini loss induces highly-discriminative features, it does not ensure that the distribution of the high-level features matches the distribution of the classes. As such, we introduce another loss term to minimize the Kullback–Leibler divergence between the two distributions. We conduct experiments on two image classification data sets (CIFAR-100 and Caltech 101), considering multiple neural architectures ranging from convolutional networks (ResNet-17, ResNet-18, ResNet-50) to transformers (CvT). Our empirical results show that integrating our novel loss terms into the training objective consistently out-performs the models trained with cross-entropy alone, with-out increasing the inference time at all.
One-Stage Object Referring with Gaze Estimation

The classic object referring task aims at localizing the referred object in the image and requires a reference image and a natural language description as inputs. Given the facts that gaze signal can be easily obtained by a modern human-computer interaction system with a camera and that human tends to look at the object when referring to it, we propose a novel gaze-assisted object referring framework. The formulation not only simplifies the state-of-the-art gaze-assisted object referring system requiring many input signals besides gaze, but also incorporates the one-stage object detection idea to improve the inference efficiency. More importantly, it implicitly considers all object candidates and thus resolves the main pain point of existing two-stage object referring solutions for proposing an appropriate number of candidates – it cannot be too large, otherwise the computational cost can be prohibitive; it cannot be too small, otherwise the chance of missing a referred object can be significant. To utilize the gaze information, we propose to build a gaze heatmap by using the anchor position encoding map and the gaze prediction result. The gaze heatmap and the language feature are then merged into the feature pyramid in the object detection as the final one-stage referring system. In the CityScapes-OR dataset, the proposed method outperforms the state-of-the-art by 7.8% for Acc@1.
Integrating Pose and Mask Predictions for Multi-person in Videos

In real-world applications for video editing, humans are arguably the most important objects. When editing videos of humans, the efficient tracking of fine-grained masks and body joints is the fundamental requirement. In this paper, we propose a simple and efficient system for jointly tracking pose and segmenting high-quality masks for all humans in the video. We design a pipeline that globally tracks pose and locally segments fine-grained masks. Specifically, CenterTrack is first employed to track human poses by viewing the whole scene, and then the proposed local segmentation network leverages the pose information as a powerful query to carry out high-quality segmentation. Furthermore, we adopt a highly light-weight MLP-Mixer layer within the segmentation network that can efficiently propagate the query pose throughout the region of interest with minimal overhead. For the evaluation, we collect a new benchmark called KineMask which includes various appearances and actions. The experimental results demonstrate that our method has superior fine-grained segmentation performance. Moreover, it runs at 33 fps, achieving a great balance of speed and accuracy compared to the prevailing online Video Instance Segmentation methods.
Exploiting Distortion Information for Multi-degraded Image Restoration

In recent years, tremendous studies have been performed on the image distortion restoration task and deep learning-based methods have shown prominent performance improvement. However, assuming only a single distortion to an image may not be applicable in many real-world scenarios. To mitigate the issue, some studies have proposed multi-distortion datasets by applying the corruptions sequentially or spatially. In this work, we integrate the two perspectives on the multi-distortion nature and propose a new dataset that is a holistic multi-distortion dataset. To restore the multi-distortion effectively, we introduce a distortion information-guided restoration network, which exploits the conditional distortion information when reconstructing a given image. To do that, our framework first predicts the distortion type and their strength and delivers these to the restoration module. In our experiments, we show that the proposed model exceeds the others and we also demonstrate that any backbone network benefits from receiving the distortion information as prior knowledge.
RODD: A Self-Supervised Approach for Robust Out-of-Distribution Detection

Recent studies have started to address the concern of detecting and rejecting the out-of-distribution (OOD) samples as a major challenge in the safe deployment of deep learning (DL) models. It is desired that the DL model should only be confident about the in-distribution (ID) data which reinforces the driving principle of the OOD detection. In this paper, we propose a simple yet effective generalized OOD detection method independent of out-of-distribution datasets. Our approach relies on self-supervised feature learning of the training samples, where the embeddings lie on a compact low-dimensional space. Motivated by the recent studies that show self-supervised adversarial contrastive learning helps robustify the model, we empirically show that a pre-trained model with self-supervised contrastive learning yields a better model for uni-dimensional feature learning in the latent space. The method proposed in this work, referred to as RODD, outperforms SOTA detection performance on extensive suite of benchmark datasets on OOD detection tasks. On the CIFAR-100 benchmarks, RODD achieves a 26.97 % lower false positive rate (FPR@95) compared to SOTA methods. Our code is publicly available. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
SpiderNet: Hybrid Differentiable-Evolutionary Architecture Search via Train-Free Metrics

Neural Architecture Search (NAS) algorithms are intended to remove the burden of manual neural network design, and have shown to be capable of designing excellent models for a variety of well-known problems. However, these algorithms require a variety of design parameters in the form of user configuration or hard-coded decisions which limit the variety of networks that can be discovered. This means that NAS algorithms do not eliminate model design tuning, they instead merely shift the burden of where that tuning needs to be applied. In this paper, we present SpiderNet, a hybrid differentiable-evolutionary and hardware-aware algorithm that rapidly and efficiently produces state-of-the-art networks. More importantly, SpiderNet is a proof-of-concept of a minimally-configured NAS algorithm; the majority of design choices seen in other algorithms are incorporated into SpiderNet’s dynamically-evolving search space, minimizing the number of user choices to just two: reduction cell count and initial channel count. SpiderNet produces models highly-competitive with the state-of-the-art, and outperforms random search in accuracy, runtime, memory size, and parameter count.
Information Elevation Network for Online Action Detection and Anticipation

Given a partially observed video segment, online action detection and anticipation aim to identify a current action and forecast future actions, respectively. To detect actions in a streaming video for monitoring applications including surveillance, robot assistants, and autonomous driving, online action detection methods have been proposed. Considering the importance of current action in online action detection, we introduce a novel information elevation unit (IEU) that lifts and accumulates the past information relevant to the current action, to compensate for forgotten essential information. Using the IEUs, we propose an information elevation network (IEN) that effectively identifies a current action and anticipates future actions through the dense prediction of past and current action classes within the video segment. For its practical use in online monitoring applications, our IEN takes visual features extracted from a fast action recognition using only RGB frames because extracting optical flows requires heavy computation overhead. On THUMOS-14 and TVSeries, our IEN out-performs state-of-the-art methods using only RGB frames. Furthermore, on the THUMOS-14 dataset, our IEN outperforms the state-of-the-art methods.
Z-Domain Entropy Adaptable Flex for Semi-supervised Action Recognition in the Dark

The subtask of Human Action Recognition (AR) in the dark is gaining a lot of traction nowadays, which takes a significant place in the field of computer vision. The implementation of its application includes self-driving at night, human-pose estimation, night surveillance, etc. Currently, solutions such as DLN for AR have emerged. However, due to the poor accuracy even when leveraging on large amounts of datasets and complex architectures, the development of AR in the dark has been slow to progress. In this paper, we propose a novel and straightforward method: Z-Domain Entropy Adaptable Flex. This constructs a neural network architecture R(2+1)D, including (i) a self-attention mechanism, which combines and extracts corresponding and complementary features from the dual pathways; (ii) Zero-DCE low light image enhancement, which improves enhanced quality; and (iii) FlexMatch method, which can generates the pseudo-labels flexibly. With the help of pseudo-labels from FlexMatch, our proposed Z-DEAF method facilitates the process of gaining desired classification boundaries. This works by repeating Expanding Entropy and Shrinking Entropy. It aims to solve the problem of unclear classification boundaries between the categories. Our model obtains superior performance in experiments, and achieves state-of-the-art results on ARID.
Where did I leave my keys? — Episodic-Memory-Based Question Answering on Egocentric Videos

Humans have a remarkable ability to organize, compress and retrieve episodic memories throughout their daily life. Current AI systems, however, lack comparable capabilities as they are mostly constrained to an analysis with access to the raw input sequence, assuming an unlimited amount of data storage which is not feasible in realistic deployment scenarios. For instance, existing Video Question Answering (VideoQA) models typically reason over the video while already being aware of the question, thus requiring to store the complete video in case the question is not known in advance.In this paper, we address this challenge with three main contributions: First, we propose the Episodic Memory Question Answering (EMQA) task as a specialization of VideoQA. Specifically, EMQA models are constrained to keep only a constant-sized representation of the video input, thus automatically limiting the computation requirements at query time. Second, we introduce a new egocentric VideoQA dataset called QaEgo4D, far larger than existing egocentric VideoQA datasets and featuring video length unprecedented in VideoQA datasets in general. Third, we present extensive experiments on the new dataset, comparing various baseline models in both the VideoQA and the EMQA setting. To facilitate future research on egocentric VideoQA as well as episodic memory representation and retrieval, we publish our code and dataset.
Multi-label Learning with User Credit Data in China Based on MLKNN

Aiming at the problem of numerous variables in credit data, a large amount of sample data, and inability to intuitively reflect user portraits. This paper uses the MLKNN algorithm to perform multi-label learning on the credit data. According to the results of the algorithm training under 24 sets of k values, the optimal number of neighbor samples of the algorithm on the sample set is 14. On this basis, this paper further analyzes The general portrait of credit users in my country has the following characteristics: more than 50% of credit users are users with stable personal development, low frequency of credit activities, and low-to-medium attention to credit status. Meanwhile, we find that the users with higher frequency of credit activities pay more attention to credit status. This research can provide some reference for commercial banks or other financial institutions in lending and credit management.
Anomaly Detection of Semiconductor Processing Data Based on DTW-LOF Algorithm

Fault detection and classification (FDC) of equipment sensor data is of great significance in semiconductor wafer manufacturing to monitor equipment and the processing wafer condition. The early detections of anomalies of sensor data facilitate later process adjustments and avoid further economic loss. Sensor data varies greatly among different processing wafer lots because of machine status changes during the maintenance cycles. This paper combines the Dynamic Time Warping (DTW) and Local Outlier Factor (LOF) algorithms to achieve stable anomaly detection of sensor data under unstable machine conditions. The DTW-LOF model shows good anomaly detection accuracy in different chip processing technologies with a small amount of data. Since the anomalies of semiconductor processing were usually reflected as comprehensive abnormality of multi-sensor in same time ranges, the normalized DTW distance of these sensors could effectively identify the processing abnormal stage and sensors.
Fall Detection Using Self-Supervised Pre-Training Model

Several researchers have developed fall detection using wearable sensors due to their flexibility and nature of privacy. Most of those developed methods are supervised deep learning methods. However, data annotation is expensive because we use camera video recording and playback of each participant’s recorded video to label the data. This paper presents how to use unlabeled data to pre-train FCN and ResNet models, and use labeled data to fine-tune those pre-trained weights. We collected unlabeled and labeled data and applied self-supervised learning to detect falls. The experiment in this study suggested that the best performance can be achieved by using pre-trained weights of unlabeled data from the accelerometer and gyroscope sensors. Furthermore, oversampling and modified loss functions are used to handle the dataset’s imbalance classes. With the ResNet pre-trained weights and training using the labeled data, the experiments achieved an F1Score of 0.98.
FAIVconf: Face Enhancement for AI-Based Video Conference with Low Bit-Rate

Recently, high-quality video conferencing with fewer transmission bits becomes a very hot and challenging problem. We propose FAIVConf, a specially designed video compression framework for video conferencing, based on the effective neural human face generation techniques. FAIVConf brings together several designs to improve the system robustness in real video conference scenarios: face swapping to avoid artifacts in background animation; facial blurring to decrease transmission bit-rate and maintain quality of extracted facial landmarks; and dynamic source update for face view interpolation to accommodate a large range of head poses. Our method achieves significant bit-rate reduction in video conference and gives much better visual quality under the same bit-rate compared with H.264 and H.265 coding schemes.
Multi-Augmentation for Efficient Self-Supervised Visual Representation Learning

In recent years, self-supervised learning has been studied to deal with the limitation of available labeled-dataset. Among the major components of self-supervised learning, the data augmentation pipeline is one key factor in enhancing the resulting performance. However, most researchers manually designed the augmentation pipeline, and the limited collections of transformation may cause the lack of robustness of the learned feature representation. In this work, we proposed Multi-Augmentations for Self-Supervised Representation Learning (MA-SSRL), which fully searched for various augmentation policies to build the entire pipeline to improve the robustness of the learned feature representation. MA-SSRL successfully learns the invariant feature representation and presents an efficient, effective, and adaptable data augmentation pipeline for self-supervised pre-training on different distribution and domain datasets. MA-SSRL outperforms the previous state-of-the-art methods on transfer and semi-supervised benchmarks while requiring fewer training epochs. Code available on GitHub <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Pyramid-Context Guided Feature Fusion for RGB-D Semantic Segmentation

Incorporating depth information into RGB images has proven its effectiveness in semantic segmentation. The multi-modal feature fusion, which integrates depth and RGB features, is a crucial component determining segmentation accuracy. Most existing multi-modal feature fusion schemes enhance multi-modal features via channel-wise attention modules which leverage global context information. In this work, we propose a novel pyramid-context guided fusion (PCGF) module to fully exploit the complementary information from the depth and RGB features. The proposed PCGF utilizes both local and global contexts inside the attention module to provide effective guidance for fusing cross-modal features of inconsistent semantics. Moreover, we introduce a lightweight yet practical multi-level general fusion module to combine the features at multiple levels of abstraction to enable high-resolution prediction. Utilizing the proposed feature fusion modules, our Pyramid-Context Guided Network (PCGNet) can learn discriminative features by taking full advantage of multi-modal and multi-level information. Our comprehensive experiments demonstrate that the proposed PCGNet achieves state-of-the-art performance on two benchmark datasets NYUDv2 and SUN-RGBD.
Interaction Guided Hand-Held Object Detection

General object detection methods detect various objects independently without paying attention to the relationship between objects. Independent detection of hands and hand-held objects does not take the hand-object relation into account, so the object detector does not achieve optimal performance in such scenes with associated objects. The detection of hand-held objects is an important topic for its wide applications, but few works have been done to model this specific scenario. In this paper, we uses information about the interaction between the hand and the hand-held object as well as causal information when observing the hand-held object as a person to build a novel hand-held object detection model. Experimental results show that our method greatly improves hand-held object detection performance.