Reviewerly/exHarmony huggingface dataset:
Efficient Adversarial Attacks for Visual Object Tracking

Visual object tracking is an important task that requires the tracker to find the objects quickly and accurately. The existing state-of-the-art object trackers, i.e., Siamese based trackers, use DNNs to attain high accuracy. However, the robustness of visual tracking models is seldom explored. In this paper, we analyze the weakness of object trackers based on the Siamese network and then extend adversarial examples to visual object tracking. We present an end-to-end network FAN (Fast Attack Network) that uses a novel drift loss combined with the embedded feature loss to attack the Siamese network based trackers. Under a single GPU, FAN is efficient in the training speed and has a strong attack performance. The FAN can generate an adversarial example at 10ms, achieve effective targeted attack (at least 40% drop rate on OTB) and untargeted attack (at least 70% drop rate on OTB).
Learning from Scale-Invariant Examples for Domain Adaptation in Semantic Segmentation

Self-supervised learning approaches for unsupervised domain adaptation (UDA) of semantic segmentation models suffer from challenges of predicting and selecting reasonable good quality pseudo labels. In this paper, we propose a novel approach of exploiting scale-invariance property of the semantic segmentation model for self-supervised domain adaptation. Our algorithm is based on a reasonable assumption that, in general, regardless of the size of the object and stuff (given context) the semantic labeling should be unchanged. We show that this constraint is violated over the images of the target domain, and hence could be used to transfer labels in-between differently scaled patches. Specifically, we show that semantic segmentation model produces output with high entropy when presented with scaled-up patches of target domain, in comparison to when presented original size images. These scale-invariant examples are extracted from the most confident images of the target domain. Dynamic class specific entropy thresholding mechanism is presented to filter out unreliable pseudo-labels. Furthermore, we also incorporate the focal loss to tackle the problem of class imbalance in self-supervised learning. Extensive experiments have been performed, and results indicate that exploiting the scale-invariant labeling, we outperform existing self-supervised based state-of-the-art domain adaptation methods. Specifically, we achieve 1.3% and 3.8% of lead for GTA5 to Cityscapes and SYNTHIA to Cityscapes with VGG16-FCN8 baseline network.
Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems

As the complexity of Deep Neural Network (DNN) models increases, their deployment on mobile devices becomes increasingly challenging, especially in complex vision tasks such as image classification. Many of recent contributions aim either to produce compact models matching the limited computing capabilities of mobile devices or to offload the execution of such burdensome models to a compute-capable device at the network edge - the edge servers. In this paper, we propose to modify the structure and training process of DNN models for complex image classification tasks to achieve in-network compression in the early network layers. Our training process stems from knowledge distillation, a technique that has been traditionally used to build small - student - models mimicking the output of larger - teacher - models. Here, we adopt this idea to obtain aggressive compression while preserving accuracy. Our results demonstrate that our approach is effective for state-of-the-art models trained over complex datasets, and can extend the parameter region in which edge computing is a viable and advantageous option. Additionally, we demonstrate that in many settings of practical interest we reduce the inference time with respect to specialized models such as MobileNet v2 executed at the mobile device, while improving accuracy.
Online Invariance Selection for Local Feature Descriptors

To be invariant, or not to be invariant: that is the question formulated in this work about local descriptors. A limitation of current feature descriptors is the trade-off between generalization and discriminative power: more invariance means less informative descriptors. We propose to overcome this limitation with a disentanglement of invariance in local descriptors and with an online selection of the most appropriate invariance given the context. Our framework ( https://github.com/rpautrat/LISRD ) consists in a joint learning of multiple local descriptors with different levels of invariance and of meta descriptors encoding the regional variations of an image. The similarity of these meta descriptors across images is used to select the right invariance when matching the local descriptors. Our approach, named Local Invariance Selection at Runtime for Descriptors (LISRD), enables descriptors to adapt to adverse changes in images, while remaining discriminative when invariance is not required. We demonstrate that our method can boost the performance of current descriptors and outperforms state-of-the-art descriptors in several matching tasks, when evaluated on challenging datasets with day-night illumination as well as viewpoint changes.
Context-Aware RCNN: A Baseline for Action Detection in Videos

Video action detection approaches usually conduct actor-centric action recognition over RoI-pooled features following the standard pipeline of Faster-RCNN. In this work, we first empirically find the recognition accuracy is highly correlated with the bounding box size of an actor, and thus higher resolution of actors contributes to better performance. However, video models require dense sampling in time to achieve accurate recognition. To fit in GPU memory, the frames to backbone network must be kept low-resolution, resulting in a coarse feature map in RoI-Pooling layer. Thus, we revisit RCNN for actor-centric action recognition via cropping and resizing image patches around actors before feature extraction with I3D deep network. Moreover, we found that expanding actor bounding boxes slightly and fusing the context features can further boost the performance. Consequently, we develop a surprisingly effective baseline (Context-Aware RCNN) and it achieves new state-of-the-art results on two challenging action detection benchmarks of AVA and JHMDB. Our observations challenge the conventional wisdom of RoI-Pooling based pipeline and encourage researchers rethink the importance of resolution in actor-centric action recognition. Our approach can serve as a strong baseline for video action detection and is expected to inspire new ideas for this filed. The code is available at https://github.com/MCG-NJU/CRCNN-Action .
Adaptive Offline Quintuplet Loss for Image-Text Matching

Existing image-text matching approaches typically leverage triplet loss with online hard negatives to train the model. For each image or text anchor in a training mini-batch, the model is trained to distinguish between a positive and the most confusing negative of the anchor mined from the mini-batch (i.e. online hard negative). This strategy improves the model’s capacity to discover fine-grained correspondences and non-correspondences between image and text inputs. However, the above approach has the following drawbacks: (1) the negative selection strategy still provides limited chances for the model to learn from very hard-to-distinguish cases. (2) The trained model has weak generalization capability from the training set to the testing set. (3) The penalty lacks hierarchy and adaptiveness for hard negatives with different “hardness” degrees. In this paper, we propose solutions by sampling negatives offline from the whole training set. It provides “harder” offline negatives than online hard negatives for the model to distinguish. Based on the offline hard negatives, a quintuplet loss is proposed to improve the model’s generalization capability to distinguish positives and negatives. In addition, a novel loss function that combines the knowledge of positives, offline hard negatives and online hard negatives is created. It leverages offline hard negatives as the intermediary to adaptively penalize them based on their distance relations to the anchor. We evaluate the proposed training approach on three state-of-the-art image-text models on the MS-COCO and Flickr30K datasets. Significant performance improvements are observed for all the models, proving the effectiveness and generality of our approach. Code is available at https://github.com/sunnychencool/AOQ .
Self-filtering image dehazing with self-supporting module

As a pre-processing step of computer vision applications, single image dehazing remains challenging due to existing inefficiencies in the restoration of content and details. In this paper, the self-supporting dehazing network (SSDN) is proposed to overcome these two problems. For the restoration of image content, the self-filtering block is introduced to remove redundant features, hence improving the representation abilities of learned features. For the recovery of image details, a novel self-supporting module is proposed as a crucial component of the proposed SSDN. With this module, the complementary information among support images that are transformed from multi-level features is explored. By incorporating such information, the self-supporting module can learn more intrinsic image characteristics and generate fine-detail images. Experimental results demonstrate that the proposed SSDN outperforms state-of-the-art dehazing methods in terms of both quantitative accuracy and qualitative visual effect.
Robust Re-Identification by Multiple Views Knowledge Distillation

To achieve robustness in Re-Identification, standard methods leverage tracking information in a Video-To-Video fashion. However, these solutions face a large drop in performance for single image queries (e.g., Image-To-Video setting). Recent works address this severe degradation by transferring temporal information from a Video-based network to an Image-based one. In this work, we devise a training strategy that allows the transfer of a superior knowledge, arising from a set of views depicting the target object. Our proposal – Views Knowledge Distillation (VKD) – pins this visual variety as a supervision signal within a teacher-student framework, where the teacher educates a student who observes fewer views. As a result, the student outperforms not only its teacher but also the current state-of-the-art in Image-To-Video by a wide margin (6.3% mAP on MARS, 8.6% on Duke and 5% on VeRi-776). A thorough analysis – on Person, Vehicle and Animal Re-ID – investigates the properties of VKD from a qualitatively and quantitatively perspective. Code is available at https://github.com/aimagelab/VKD .
DAPath: Distance-aware knowledge graph reasoning based on deep reinforcement learning

Knowledge graph reasoning aims to find reasoning paths for relations over incomplete knowledge graphs (KG). Prior works may not take into account that the rewards for each position (vertex in the graph) may be different. We propose the distance-aware reward in the reinforcement learning framework to assign different rewards for different positions. We observe that KG embeddings are learned from independent triples and therefore cannot fully cover the information described in the local neighborhood. To this effect, we integrate a graph self-attention (GSA) mechanism to capture more comprehensive entity information from the neighboring entities and relations. To let the model remember the path, we incorporate the GSA mechanism with GRU to consider the memory of relations in the path. Our approach can train the agent in one-pass, thus eliminating the pre-training or fine-tuning process, which significantly reduces the problem complexity. Experimental results demonstrate the effectiveness of our method. We found that our model can mine more balanced paths for each relation.
Convex Non-Negative Matrix Factorization With Adaptive Graph for Unsupervised Feature Selection

Unsupervised feature selection (UFS) aims to remove the redundant information and select the most representative feature subset from the original data, so it occupies a core position for high-dimensional data preprocessing. Many proposed approaches use self-expression to explore the correlation between the data samples or use pseudolabel matrix learning to learn the mapping between the data and labels. Furthermore, the existing methods have tried to add constraints to either of these two modules to reduce the redundancy, but no prior literature embeds them into a joint model to select the most representative features by the computed top ranking scores. To address the aforementioned issue, this article presents a novel UFS method via a convex non-negative matrix factorization with an adaptive graph constraint (CNAFS). Through convex matrix factorization with adaptive graph constraint, it can dig up the correlation between the data and keep the local manifold structure of the data. To our knowledge, it is the first work that integrates pseudo label matrix learning into the self-expression module and optimizes them simultaneously for the UFS solution. Besides, two different manifold regularizations are constructed for the pseudolabel matrix and the encoding matrix to keep the local geometrical structure. Eventually, extensive experiments on the benchmark datasets are conducted to prove the effectiveness of our method. The source code is available at: https://github.com/misteru/CNAFS.
K-DBSCAN: An improved DBSCAN algorithm for big data

Big data storage and processing are among the most important challenges now. Among data mining algorithms, DBSCAN is a common clustering method. One of the most important drawbacks of this algorithm is its low execution speed. This study aims to accelerate the DBSCAN execution speed so that the algorithm can respond to big datasets in an acceptable period of time. To overcome the problem, an initial grouping was applied to the data in this article through the K-means++ algorithm. DBSCAN was then employed to perform clustering in each group separately. As a result, the computational burden of DBSCAN execution reduced and the clustering execution speed increased significantly. Finally, border clusters were merged if necessary. According to the results of executing the proposed algorithm, it managed to greatly reduce the DBSCAN execution time (98% in the best-case scenario) with no significant changes in the qualitative evaluation criteria for clustering.
Blockchain-Based Energy Trading and Load Balancing Using Contract Theory and Reputation in a Smart Community

The rapid deployment of Electric Vehicles (EVs) and the integration of renewable energy sources have ameliorated the existing power systems and contributed to the development of greener smart communities. However, load balancing problems, security threats, privacy leakage issues, etc., remain unresolved. Many blockchain-based approaches have been used in literature to solve the aforementioned challenges. However, they are not sufficient to obtain satisfactory results because of the inefficient energy management methods and time-intensiveness of the primitive cryptographic executions on the network devices. In this paper, an efficient and secure blockchain-based Energy Trading (ET) model is proposed. It leverages the contract theory, incentive mechanism, and a reputation system for information asymmetry scenario. In order to motivate the ET entities to trade energy locally and EVs to participate in smart energy management, the proposed incentive provisioning mechanism plays a vital role. Besides, a reputation system improves the reliability and efficiency of the system and discourages the blockchain nodes from acting maliciously. A novel consensus algorithm, i.e., Proof of Work based on Reputation (PoWR), is proposed to reduce transaction confirmation latency and block creation time. Moreover, a shortest route algorithm, i.e., the Dijkstra algorithm, is implemented in order to reduce the traveling distance and energy consumption of the EVs during ET. The performance of the proposed model is evaluated using peak to average ratio, social welfare, utility of local aggregator, etc., as performance metrics. Moreover, privacy and security analyses of the system are also presented.
Referring Expression Comprehension: A Survey of Methods and Datasets

Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It is more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In this survey, we first examine the state-of-the-art by comparing modern approaches to the problem. We classify methods by their mechanism to encode the visual and textual modalities. In particular, we examine the common approach of joint embedding images and expressions to a common feature space. We also discuss modular architectures and graph-based models that interface with structured graph representation. In the second part of this survey, we review the datasets available for training and evaluating REC approaches. We then group results according to the datasets, backbone models, settings so that they can be fairly compared. Finally, we discuss promising future directions for this field, in particular the compositional referring expression comprehension that requires more reasoning steps to address.
High dimensional very short-term solar power forecasting based on a data-driven heuristic method

Improving the accuracy of solar power forecasting has become crucial for dealing with the negative effects of the integration of continually increasing solar power into power systems. This is a more challenging task when historical solar radiation data has not been recorded and no specific sky imaging equipment is available. This paper proposes a univariate data-driven method to improve the accuracy of very short-term electrical solar power forecasting. This approach includes defining new features that efficiently tackle the nonlinear characteristics of electrical solar power. An instance-based variable selection is also used to identify the best relevant features. Three state-of-the-art learning algorithms (i.e. neural networks, support vector regression, and random forest) have been used and compared as prediction algorithms of the proposed method. The effectiveness of the proposed approach is evaluated in a 15-min ahead prediction trial using a real solar power dataset and against three evaluation measures. The results show the proposed method significantly enhances the performance of very short-term solar power forecasting.
Entropy Maximization and Meta Classification for Out-of-Distribution Detection in Semantic Segmentation

Deep neural networks (DNNs) for the semantic segmentation of images are usually trained to operate on a predefined closed set of object classes. This is in contrast to the "open world" setting where DNNs are envisioned to be deployed to. From a functional safety point of view, the ability to detect so-called "out-of-distribution" (OoD) samples, i.e., objects outside of a DNN’s semantic space, is crucial for many applications such as automated driving. A natural baseline approach to OoD detection is to threshold on the pixel-wise softmax entropy. We present a two-step procedure that significantly improves that approach. Firstly, we utilize samples from the COCO dataset as OoD proxy and introduce a second training objective to maximize the softmax entropy on these samples. Starting from pretrained semantic segmentation networks we re-train a number of DNNs on different in-distribution datasets and consistently observe improved OoD detection performance when evaluating on completely disjoint OoD datasets. Secondly, we perform a transparent post-processing step to discard false positive OoD samples by so-called "meta classification." To this end, we apply linear models to a set of hand-crafted metrics derived from the DNN’s softmax probabilities. In our experiments we consistently observe a clear additional gain in OoD detection performance, cutting down the number of detection errors by 52% when comparing the best baseline with our results. We achieve this improvement sacrificing only marginally in original segmentation performance. Therefore, our method contributes to safer DNNs with more reliable overall system performance.
Improved Chaos-Based Cryptosystem for Medical Image Encryption and Decryption

In the medical sector, the digital image is multimedia data that contain secret information. However, designing an efficient secure cryptosystem to protect the confidential images in sharing is a challenge. In this work, we propose an improved chaos-based cryptosystem to encrypt and decrypt rapidly secret medical images. A complex chaos-based PRNG is suggested to generate a high-quality key that presents high randomness behaviour, high entropy, and high complexity. An improved architecture is proposed to encrypt the secret image that is based on permutation, substitution, and diffusion properties. In the first step, the image’s pixels are randomly permuted through a matrix generated using the PRNG. Next, pixel’s bits are permuted using an internal condition. After that, the pixels are substituted using two different S-boxes with an internal condition. In the final step, the image is diffused by XORing pixels with the key stream generated by the PRNG in order to acquire an encrypted image. R rounds of encryption can be performed in a loop to increase the complexity. The cryptosystem is evaluated in depth by his application on several medical images with different types, contents, and sizes. The obtained simulation results demonstrate that the system enables high-level security and performance. The information entropy of the encrypted image has achieved an average of 7.9998 which is the most important feature of randomness. The algorithm can take full advantage of parallelism and pipeline execution in hardware implementation to meet real-time requirements. The PRNG was tested by NIST 800-22 test suit, which indicates that it is suitable for secure image encryption. It provides a large key space of 2192 which resists the brute-force attack. However, the cryptosystem is appropriate for medical image securing.