Reviewerly/exHarmony huggingface dataset:
Graph Convolutional Networks and Manifold Ranking for Multimodal Video Retrieval

Despite the impressive advances obtained by supervised deep learning approaches on retrieval and classification tasks, how to acquire labeled data for training remains a challenging bottleneck. In this scenario, the need for developing more effective content-based retrieval approaches capable of taking advantage of multimodal information and advances in unsupervised learning becomes imperative. Based on such observations, we propose two novel approaches that combine Graph Convolutional Networks (GCNs) with rank-based manifold learning methods. The GCN models were trained in an unsupervised way, using the Deep Graph Infomax algorithm, and the proposed approaches employ recent rank-based manifold learning methods. Multimodal information is exploited through pre-trained CNNs via transfer learning for extracting audio, image, and video features. The proposed approaches were evaluated on three public action recognition datasets. High-effective results were obtained, reaching relative gains up to +29.44% of MAP compared to baseline approaches without GCNs. The experimental evaluation also considered classical and recent baselines in the literature.
Faketracer: Exposing Deepfakes with Training Data Contamination

We describe a proactive defense method to expose Deep-Fakes with training data contamination. Note that the existing methods usually focus on defending from general DeepFakes, which are synthesized by GAN using random noise. In contrast, our method is dedicated to defending from native Deep-Fakes, which is synthesized by auto-encoder that involves face swapping and encoding-decoding process that general DeepFakes do not have. Specifically, we design two types of traces namely sustainable traces and erasable traces, which are added on the faces to manipulate the training of DeepFake models. Consequently, the trained DeepFake model can synthesize faces with sustainable traces but no erasable traces. With the help of these two traces, we can expose DeepFakes proactively. Our method is compared with recent passive and proactive defense methods, which corroborates the efficacy of our method.
Local Embedding for Axial Attention

Recently, the researches on deep neural networks using self-attention have been actively conducted and shown to be effective. However, self-attention requires a large amount of computational cost. Axial Attention reduces the computational complexity by factorizing 2D self-attention into two 1D self-attentions, but the visualization of activation region was found to be inaccurate compared to CNN. The inaccurate of activation region means that unrelated regions to true class are referred for classification. It will decrease the accuracy of segmentation, which is a pixel-by-pixel classification, in identifying object boundaries. In this study, we attempted to reduce the problem by introducing a Local Embedding Unit to Axial Attention. In addition, by improving the structure of Axial Attention, we were able to improve the accuracy while reducing the computational cost and the number of parameters. Our model achieved useful results on the classification of ImageNet and segmentation of the CamVid dataset.
Anisotropic Edge Detection in Catadioptric Images

Catadioptric images are produced in omnidirectional vision systems and can be expressed on Riemannian manifolds. The existing edge detectors are operated either in Euclidean space, or on Riemannian manifolds with isotropic image filtering. In this paper, a new type of edge detection is proposed—it is operated on Riemannian manifolds with anisotropic image filtering. For that, an anisotropic image filtering kernel on Riemannian manifolds is derived by solving the anisotropic heat equation embedded with Riemannian metric. With this kernel, a novel anisotropic edge detector is then developed. Compared to an edge detector operated in Euclidean space, our edge detector is more suitable for catadioptric images, since their geometric structure information will be taken into account in the detection process. Compared to existing edge detectors customized for catadioptric images, the new edge detector has a higher efficiency in preserving image edges and thus can produce more true positives.
MVMO: A Multi-Object Dataset for Wide Baseline Multi-View Semantic Segmentation

We present MVMO (Multi-View, Multi-Object dataset): a synthetic dataset of 116,000 scenes containing randomly placed objects of 10 distinct classes and captured from 25 camera locations in the upper hemisphere. MVMO comprises photorealistic, path-traced image renders, together with semantic segmentation ground truth for every view. Unlike existing multi-view datasets, MVMO features wide baselines between cameras and high density of objects, which lead to large disparities, heavy occlusions and view-dependent object appearance. Single view semantic segmentation is hindered by self and inter-object occlusions that could benefit from additional viewpoints. Therefore, we expect that MVMO will propel research in multi-view semantic segmentation and cross-view semantic transfer. We also provide baselines that show that new research is needed in such fields to exploit the complementary information of multi-view setups <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Efficient Feature Compression for the Object Tracking Task

In object tracking systems, often clients capture video, encode it and transmit it to a server that performs the actual machine task. In this paper we propose an alternative architecture, where we instead transmit features to the server. Specifically, we partition the Joint Detection and Embedding (JDE) person tracking network into client and server side sub-networks and code the intermediate tensors i.e. features. The features are compressed for transmission using a Deep Neural Network (DNN) we design and train specifically for carrying out the tracking task. The DNN uses trainable non-uniform quantizers, conditional probability estimators, hierarchical coding; concepts that have been used in the past for neural networks based image and video compression. Additionally, the DNN includes a novel parameterized dual-path layer that comprises of an autoencoder in one path and a convolution layer in the other. The tensor output by each path is added before being consumed by subsequent layers. The parameter value for this dual-path layer controls the output channel count and correspondingly the bitrate of transmitted bitstream. We demonstrate that our model improves coding efficiency by 43.67% over state-of-the-art Versatile Video Coding standard that codes the source video in pixel domain.
A Cognitive Perspective on Subjective and Objective Diagnostic Image Quality Models

We present a new perspective on diagnostic image quality (DIQ) for mammography (MG) aimed at future clinical decision support applications. We posit that DIQ is based on three interdependent criteria: adequacy, visual quality, and interpretability. Adequacy is addressed by imaging protocols and technique, while visual quality and interpretability ensures that human perception will be able to detect diagnostically relevant content, and both enable cognitive interpretation at a conceptual level. We have implemented a visual interface for radiologists to enter annotated scores for the three criteria and overall DIQ. Score annotations may be used to identify relevant computable image features by text mining. The image features allow building DIQ metrics potentially useful during image acquisition and reading. We have conducted preliminary research on a set of relevant image features and present initial results on their discriminative power and emerging challenges. We also discuss next steps and strategies for future research.
Fast Semantic Image Segmentation for Autonomous Systems

Fast semantic image segmentation is crucial for autonomous systems, as it allows an autonomous system (e.g., self-driving car, drone, etc.) to interpret its environment on-the-fly and decide on necessary actions by exploiting dense semantic maps. The speed of semantic segmentation on embedded computational hardware is as important as its accuracy. Thus, this paper proposes a novel framework for semantic image segmentation that is both fast and accurate. It augments existing real-time semantic image segmentation architectures by an auxiliary, parallel neural branch that is tasked to predict semantic maps in an alternative manner by utilizing Generative Adversarial Networks (GANs). Additional attention-based neural synapses linking the two branches allow information to flow between them during both the training and the inference stage. Extensive experiments on three public datasets for autonomous driving and for aerial-perspective image analysis indicate non-negligible gains in segmentation accuracy, without compromises on inference speed.
Attention-Based Neural Network For Ill-Exposed Image Correction

The present work presents an artificial neural network architecture for the restoration of images damaged by underexposure and overexposure. The problem is relevant in computer vision applications that are applied in conditions where the limitation of the sensor prevent the scene details from being adequately represented in the captured image. This research presents an attention-based architecture composed of two convolutional neural networks, where one performs a preprocessing of the input image, while the other performs the restoration and enhancement of the degraded image. Regarding the evaluation of research results, a broad range of image quality metrics is used to assess the quality of the results produced by the model. The obtained results indicate that the proposed architecture is able to enhance images damaged by exposure heterogeneity, offering gains over state-of-art models in real data.
Eogface: Deep Face Recognition via Extensional Logits

The core of face recognition task is to learn the discriminative feature representation, which has intra-class compactness and inter-class separability. In recent years, some margin-based softmax loss functions were designed to encourage the intra-class compactness, but they neglect the inter-class separability. RegularFace were proposed to increase the inter-class separability. However, RegularFace is inefficient and memory-consumptive on large datasets with large numbers of identities. In this paper, we propose a novel method, named EogFace. It can encourage both the intra-compactness and the inter-class separability. EogFace has intuitive geometric interpretation and theoretical proof, which is easy to implement and only adds negligible computational overhead. Extensive experiments on popular benchmarks of face recognition showed the effectiveness of method over existing state-of-the-art(SOTA) algorithms. Our codes will be released soon.
Sign-OPT+: An Improved Sign Optimization Adversarial Attack

We study the hard-label adversarial attacks where model information, training data, and output score are all unknown except for the final decision to an input query. Due to the security issue, model providers usually constrain the number of queries. It is challenging for adversaries to attack a model with limited queries. Sign-OPT makes great advances in query complexity, which adopts the backtracking line-search to find the optimal search direction, meanwhile the binary search is employed to obtain the minimum-distortion adversarial example. We find that this binary search costs a huge amount of queries. This paper proposes an improved Sign-OPT, termed Sign-OPT+, to enhance query efficiency further. Instead of the binary search, at each line-search stage we directly judge whether the candidate example along the new search direction locates inside or outside the decision boundary. This judgment requires only one query to achieve the optimal search direction, significantly reducing the overall queries. Experiments tested on MNIST, CIFAR-10, and ImageNet show that our Sign-OPT+ requires fewer queries and obtains a higher success rate than the state-of-the-art including Sign-OPT. The source code is available at https://github.com/GZHU-DVL/Sign-OPT-plus.
HDR-AGAN: Ghost-Free High Dynamic Range Imaging with Attention Guided Adversarial Network

Creating a high dynamic range (HDR) image from a set of low dynamic range (LDR) images with multiple exposures is always a challenging task when there is motion among the LDR image stacks. In this paper, we propose a generative adversarial network-based method with an attention module, called HDR- AGAN, to deal with the issue caused by object motions. Given three dynamic LDR images with different exposures, the well-exposed one is treated as the reference image, while the other two images are aligned to the reference one so that two additional inputs are obtained in the proposed system. An attention module operated between the reference image and the non-reference image is employed for extracting useful features. The generated HDR image is judged by the discriminator which operates on specified image areas derived by the ghost detection module. Experimental results show that, compared to existing algorithms, our method generates HDR images with less ghost artifacts and also yields better image quality in terms of several objective metrics.
A Novel Visual Feature and Gaze Driven Egocentric Video Retargeting

Egocentric vision data has become popular due to its unique way of capturing first-person perspective. However they are lengthy, contain redundant information and visual noise caused by head movements which disrupt the story being expressed through them. This paper proposes a novel visual feature and gaze driven approach to retarget egocentric videos following the principles of cinematography. This approach is divided into two parts: activity based scene detection and performing panning and zooming to produce visually immersive videos. Firstly, visually similar frames are grouped using DCT feature matching followed by SURF descriptor matching. These groups are further refined using the gaze data to generate different scenes and transitions occurring within an activity. Secondly, the mean 2D gaze positions of scenes are used for generating panning windows enclosing 75% of the frame content. This is done for performing zoom-in and zoom-out operations in the detected scenes and transitions respectively. Our approach has been tested on the GTEA and EGTEA gaze plus datasets witnessing an average accuracy of 88.1% and 72% for sub-activity identification and obtaining an average aspect ratio similarity (ARS) score of 0.967 and 0.73; 60% and 42% SIFT similarity index (SSI) respectively. Code available on Github. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Episode Difficulty Based Sampling Method for Few-Shot Classification

Most methods in few-shot learning adopt episodic training, where classes for generating episodes are randomly sampled. Here, most of the episodes are easily solvable, i.e., the dataset in terms of the episodes becomes biased towards easy ones. In this paper, we propose a novel sampling method named Episode Difficulty Based Sampling (EDBS) that aims to remove the dataset bias in terms of episode difficulty. We define the episode difficulty to be proportional to the similarity between the classes composing the episode. Then we determine an episode as easy or hard depending on their episode difficulty and design a balanced episode dataset in terms of the difficulty. Through our EDBS, few-shot networks become less biased to the easy episodes and learn detailed features necessary for solving challenging episodes. Experiments demonstrate that our sampling method is widely applicable and achieves state-of-the-art performance in the benchmarks.
A Self-Training Weakly-Supervised Framework for Pathologist-Like Histopathological Image Analysis

The advent of artificial intelligence-based tools applied to digital pathology brings the promise of reduced workload for pathologists and enhanced patient care, not to mention medical research progress. Yet, despite its great potential, the field is hindered by the paucity of annotated histological data, a limitation for developing robust deep learning models. To reduce the number of expert annotations needed for training, we introduce a novel framework combining self-training and weakly-supervised learning that uses both annotated and unannotated data samples. Inspired by how pathologists examine biopsies, our method considers whole slide images from a bird’s eye view to roughly localize the tumor area before focusing on its features at a higher magnification level. Notwithstanding the scarcity of the dataset, the experimental results show that the proposed method outperforms models trained with annotated data only and previous works analyzing the same type of lesions, thus demonstrating the efficiency of the approach.
Fast Adaptive Self-Supervised Underwater Image Enhancement

Wavelength-dependent light absorption and scattering result in the color cast and contrast degradation, which degrades the visibility of underwater images. Deep-learning-based under-water image enhancement has made significant progress in recent years. However, existing algorithms rely heavily on massive data for training, and the difficulty of collecting massive data in real-world environments limits their applicability. To alleviate this problem, we propose a fast adaptive self-supervised underwater image enhancement method, which models the entire water domain in terms of color and contrast distribution with only a few images. Specifically, to learn the mapping of color correction and contrast enhancement, our framework contains two sub-networks: a Color Mapping Net (CLM-Net) and a Contrast Mapping Net (CTM-Net). The CLM-Net learns the color correction mapping under the constraint of gray world assumption in a self-supervised manner. Besides, the CTM-Net estimates the contrast enhancement mapping under the constraint of the dark channel prior. Thanks to adaptive self-supervised training, we can primarily alleviate the need for collecting massive data and achieve effective underwater image enhancement with only ten images. Experiments demonstrate that the proposed algorithm achieves state-of-the-art performance on four benchmark datasets.
Visual Tempo Contrastive Learning for Few-Shot Action Recognition

Few-shot action recognition aims to learn novel action classes with only a few annotated samples. This is a challenging problem because motion modeling is difficult, especially when a few training samples are available. Visual tempo, which is an essential variation factor of video semantics, characterizes the dynamic motion information in action videos. In this work, we propose a visual tempo contrastive learning framework (VTCL) to tackle the few-shot action recognition problem. Specifically, we propose a visual tempo encoding (VTE) module for visual tempo learning. The VTE module samples the same action instance at different frame sampling rates to obtain different visual tempo encoding vectors, which jointly form the features of each instance. To enhance the discriminability of visual tempo encoding vectors, we propose a visual tempo contrastive encoding (VTCE) loss to promote intra-class compactness and inter-class difference of visual tempo encoding vectors. Extensive experiments demonstrate that the proposed VTCL achieves promising results among the competing state-of-the-art methods on three few-shot action recognition datasets, including Kinetics, UCF101 and HMDB51.
Bi-Modal Compositional Network for Feature Disentanglement

The purpose of Compositional Zero-Shot Learning (CZSL) is to recognise unseen (or new) compositions of known objects (e.g. apple, banana) and its states (e.g. peeled, ripe). The CZSL poses a challenging problem as the visual features of objects and states are often difficult to isolate from their composition in images. Also, the features of a state may have wide variation across different compositions. For example, the state sliced presents different visual features in compositions, sliced potato and sliced bread. Hence, for solving CZSL problem, disentanglement of state and object features is the key challenge that we attempt to address in this paper by introducing Bi-modal Compositional Network (BiComNet). The proposed BiComNet consists of two distinct branches for state and object recognition, which helps in better disentanglement of the state and object features. We also propose a novel loss component to better facilitate the disentanglement of the state and object features. Finally, the proposed BiComNet integrates two different modalities of input data, the visual and word features, to better recognise unseen state-object compositions. Extensive experiments are carried out on the benchmark datasets MIT-States & UT-Zappos50K. Our proposed algorithm outperforms competitive methods in most count.
Mixup-Based Deep Metric Learning Approaches for Incomplete Supervision

Deep learning architectures have achieved promising results in different areas (e.g., medicine, agriculture, and security). However, using those powerful techniques in many real applications becomes challenging due to the large labeled collections required during training. Several works have pursued solutions to overcome it by proposing strategies that can learn more for less, e.g., weakly and semi-supervised learning approaches. As these approaches do not usually address memorization and sensitivity to adversarial examples, this paper presents three deep metric learning approaches combined with Mixup for incomplete-supervision scenarios. We show that some state-of-the-art approaches in metric learning might not work well in such scenarios. Moreover, the proposed approaches outperform most of them in different datasets.
A Novel Framework to Detect Irrelevant Software Requirements Based on MultiPhiLDA as the Topic Model

Noise in requirements has been known to be a defect in software requirements specifications (SRS). Detecting defects at an early stage is crucial in the process of software development. Noise can be in the form of irrelevant requirements that are included within an SRS. A previous study had attempted to detect noise in SRS, in which noise was considered as an outlier. However, the resulting method only demonstrated a moderate reliability due to the overshadowing of unique actor words by unique action words in the topic–word distribution. In this study, we propose a framework to identify irrelevant requirements based on the MultiPhiLDA method. The proposed framework distinguishes the topic–word distribution of actor words and action words as two separate topic–word distributions with two multinomial probability functions. Weights are used to maintain a proportional contribution of actor and action words. We also explore the use of two outlier detection methods, namely percentile-based outlier detection (PBOD) and angle-based outlier detection (ABOD), to distinguish irrelevant requirements from relevant requirements. The experimental results show that the proposed framework was able to exhibit better performance than previous methods. Furthermore, the use of the combination of ABOD as the outlier detection method and topic coherence as the estimation approach to determine the optimal number of topics and iterations in the proposed framework outperformed the other combinations and obtained sensitivity, specificity, F1-score, and G-mean values of 0.59, 0.65, 0.62, and 0.62, respectively.