Reviewerly/exHarmony huggingface dataset:
Boosting Black-Box Attack with Partially Transferred Conditional Adversarial Distribution

This work studies black-box adversarial attacks against deep neural networks (DNNs), where the attacker can only access the query feedback returned by the attacked DNN model, while other information such as model parameters or the training datasets are unknown. One promising approach to improve attack performance is utilizing the adversarial transferability between some white-box surrogate models and the target model (i.e., the attacked model). However, due to the possible differences on model architectures and training datasets between surrogate and target models, dubbed “surrogate biases”, the contribution of adversarial transferability to improving the attack performance may be weakened. To tackle this issue, we innovatively propose a black-box attack method by developing a novel mechanism of adversarial transferability, which is robust to the surrogate biases. The general idea is transferring partial parameters of the conditional adversarial distribution (CAD) of surrogate models, while learning the untransferred parameters based on queries to the target model, to keep the flexibility to adjust the CAD of the target model on any new benign sample. Extensive experiments on benchmark datasets and attacking against real-world API demonstrate the superior attack performance of the proposed method. The code will be available at https://github.com/Kira0096/CGATTACK.
PseudoClick: Interactive Image Segmentation with Click Imitation

The goal of click-based interactive image segmentation is to obtain precise object segmentation masks with limited user interaction, i.e., by a minimal number of user clicks. Existing methods require users to provide all the clicks: by first inspecting the segmentation mask and then providing points on mislabeled regions, iteratively. We ask the question: can our model directly predict where to click, so as to further reduce the user interaction cost? To this end, we propose PseudoClick, a generic framework that enables existing segmentation networks to propose candidate next clicks. These automatically generated clicks, termed pseudo clicks in this work, serve as an imitation of human clicks to refine the segmentation mask. We build PseudoClick on existing segmentation backbones and show how the click prediction mechanism leads to improved performance. We evaluate PseudoClick on 10 public datasets from different domains and modalities, showing that our model not only outperforms existing approaches but also demonstrates strong generalization capability in cross-domain evaluation. We obtain new state-of-the-art results on several popular benchmarks, e.g., on the Pascal dataset, our model significantly outperforms existing state-of-the-art by reducing 12.4% number of clicks to achieve 85% IoU.
Deep Learning applied to Road Accident Detection with Transfer Learning and Synthetic Images

Artificial Intelligence (AI) has driven solutions in diverse areas; one of the most prominent fields is Computer Vision (CV). Accordingly, solutions to effectively detect road accidents can be a game-changer for road safety. The focus is on Vision Zero, i.e., eliminate all traffic fatalities and severe injuries. However, when an accident occurs, response time is essential. Several countries have surveillance cameras monitoring the road network. However, real-time monitoring requires a considerable number of people with training and attention. Our solution automates this with AI, using deep learning to train a model to detect potential accidents. The CV method based on binary image classification was used, being images without accidents classified as normal and images with accidents as alarm. The biggest challenge in developing the model was obtaining images of accidents from the analyzed cameras. The solution adopted was to create synthetic images of these rare events. Several architectures of Convolutional Neural Networks were tested, and it was found that the best approach was to use transfer learning. As base models for transfer learning, the best results were EfficientNetB1 and MobileNetV2. The former for its quality in prediction and the latter for its size and execution speed. As a case study, open data images from Finnish road surveillance cameras were used, provided every ten minutes. Finally, it was found that the solution trained with EfficientNetB1 as the base model has a Mean Average Precision (mAP) of 0.89 and a Matthews Correlation Coefficient (MCC) of 0.77. The solution based on MobileNetV2 has an mAP of 0.88 and an MCC of 0.71.
Solving Invariant Generation for Unsolvable Loops

Automatically generating invariants, key to computer-aided analysis of probabilistic and deterministic programs and compiler optimisation, is a challenging open problem. Whilst the problem is in general undecidable, the goal is settled for restricted classes of loops. For the class of solvable loops, introduced by Kapur and Rodríguez-Carbonell in 2004, one can automatically compute invariants from closed-form solutions of recurrence equations that model the loop behaviour. In this paper we establish a technique for invariant synthesis for loops that are not solvable, termed unsolvable loops. Our approach automatically partitions the program variables and identifies the so-called defective variables that characterise unsolvability. We further present a novel technique that automatically synthesises polynomials, in the defective variables, that admit closed-form solutions and thus lead to polynomial loop invariants. Our implementation and experiments demonstrate both the feasibility and applicability of our approach to both deterministic and probabilistic programs.
NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledge

Novel object captioning aims at describing objects absent from training data, with the key ingredient being the provision of object vocabulary to the model. Although existing methods heavily rely on an object detection model, we view the detection step as vocabulary retrieval from an external knowledge in the form of embeddings for any object's definition from Wiktionary, where we use in the retrieval image region features learned from a transformers model. We propose an end-to-end Novel Object Captioning with Retrieved vocabulary from External Knowledge method (NOC-REK), which simultaneously learns vocabulary retrieval and caption generation, successfully describing novel objects outside of the training dataset. Furthermore, our model eliminates the requirement for model retraining by simply updating the external knowledge whenever a novel object appears. Our comprehensive experiments on held-out COCO and Nocaps datasets show that our NOCREK is considerably effective against SOTAs.
Multi-Contextual Predictions with Vision Transformer for Video Anomaly Detection

Video Anomaly Detection(VAD) has been traditionally tackled in two main methodologies: the reconstruction-based approach and the prediction-based one. As the reconstruction-based methods learn to generalize the input image, the model merely learns an identity function and strongly causes the problem called generalizing issue. On the other hand, since the prediction-based ones learn to predict a future frame given several previous frames, they are less sensitive to the generalizing issue. However, it is still uncertain if the model can learn the spatio-temporal context of a video. Our intuition is that the understanding of the spatio-temporal context of a video plays a vital role in VAD as it provides precise information on how the appearance of an event in a video clip changes. Hence, to fully exploit the context information for anomaly detection in video circumstances, we designed the transformer model with three different contextual prediction streams: masked, whole and partial. By learning to predict the missing frames of consecutive normal frames, our model can effectively learn various normality patterns in the video, which leads to a high reconstruction error at the abnormal cases that are unsuitable to the learned context. To verify the effectiveness of our approach, we assess our model on the public benchmark datasets: USCD Pedestrian 2, CUHK Avenue and ShanghaiTech and evaluate the performance with the anomaly score metric of reconstruction error. The results demonstrate that our proposed approach achieves a competitive performance compared to the existing video anomaly detection methods.
mc-BEiT: Multi-choice Discretization for Image BERT Pre-training

Image BERT pre-training with masked image modeling (MIM) becomes a popular practice to cope with self-supervised representation learning. A seminal work, BEiT, casts MIM as a classification task with a visual vocabulary, tokenizing the continuous visual signals into discrete vision tokens using a pre-learned dVAE. Despite a feasible solution, the improper discretization hinders further improvements of image pre-training. Since image discretization has no ground-truth answers, we believe that the masked patch should not be assigned with a unique token id even if a better “tokenizer” can be obtained. In this work, we introduce an improved BERT-style image pre-training method, namely mc-BEiT, which performs MIM proxy tasks towards eased and refined multi-choice training objectives. Specifically, the multi-choice supervision for the masked image patches is formed by the soft probability vectors of the discrete token ids, which are predicted by the off-the-shelf image “tokenizer” and further refined by high-level inter-patch perceptions resorting to the observation that similar patches should share their choices. Extensive experiments on classification, segmentation, and detection tasks demonstrate the superiority of our method, e.g., the pre-trained ViT-B achieves 84.1% top-1 fine-tuning accuracy on ImageNet-1K classification, 49.2% $$\text {AP}^{b}$$ and 44.0% $$\text {AP}^{m}$$ of object detection and instance segmentation on COCO, 50.8% mIOU on ADE20K semantic segmentation, outperforming the competitive counterparts. The code is available at https://github.com/lixiaotong97/mc-BEiT .
ICPR 2022: Challenge on Harvesting Raw Tables from Infographics (CHART-Infographics)

The outcomes of the third Challenge on HArvesting Raw Tables from Infographics (ICPR 2022 CHART-Infographics) are presented in this work. Recognizing charts is a difficult process which we divided into the following task: Chart Image Classification (Task 1), Text Detection and Recognition (Task 2), Text Role Classification (Task 3), Axis Analysis (Task 4), Legend Analysis (Task 5), Plot Element Detection and Classification (Task 6.a), Data Extraction (Task 6.b), and End-to-End Data Extraction (Task 7). We have provided a novel dataset for training reusing all available data from previous challenges, and we also provide a brand new testing dataset for the evaluation of submissions. Both datasets were constructed by manually annotating charts extracted from the Open Access section of the PubMed Central. A total of 9 teams registered out of which 5 submitted results for different tasks of the challenge. Many submissions are based on state-of-the-art methods from computer vision, but the final scores imply that more work will be required to solve the chart recognition problem. The data, annotation tools, and evaluation scripts have been publicly released for academic use.
Effectiveness Guided Cross-Modal Information Sharing for Aligned RGB-T Object Detection

Integrating multi-modal data can significantly increase detection performance in a complex scene by introducing additional targets' information. However, most of the existing multi-modal detectors separately extract the features from the respective modalities without regarding the correlation between the modalities. Considering the spatial correlation across different modalities for aligned multi-modal data, we attempt to exploit such correlation to share target's information across different modalities, thereby enhancing the targets' feature representation capability. To this end, in this letter, we propose an Effectiveness Guided Cross-Modal Information Sharing Network (ECISNet) for aligned multi-modal data, which can still accurately detect objects when a modality fails. Specifically, the Cross-Modal Information Sharing (CIS) module is proposed to enhance the feature extraction capability by sharing information about targets across different modalities. Afterward, considering that the failed modality may interfere with other modalities when sharing information, we designed a Modal Effectiveness Guiding (MEG) module that guides the CIS module to exclude the interference of failed modalities. Extensive experiments on three latest multi-modal detection datasets demonstrate that ECISNet outperforms relevant state-of-the-art detection algorithms.
A Novel Steganography Technique for Digital Images Using the Least Significant Bit Substitution Method

Communication has become a lot easier in this era of technology, development of highspeed computer networks, and the inexpensive uses of Internet.Therefore, data transmission has become vulnerable to and unsafe from different external attacks.Every communication body wants to secure their data while communicating over the Internet.The internet has various benefits but the main demerit is the privacy and security and the transmission of data over insecure network or channel may happen.Various techniques used for secure communication in order to address these issues, steganography plays an important role.Steganography is the process of obfuscation that makes something incomprehensible and unclear.Different image steganography research methods are proposed recently but each has their advantages and disadvantages and still have necessity to develop some better image steganography mechanisms to achieve the reliability between the basics criteria of image steganography.Therefore, the proposed work, in this paper, is based on the Least Significant Bit (LSB) substitution method.The LSB substitution method can minimize the error rate in embedding process and can achieve greater reliability in criteria, using novel algorithm based on value difference.In this paper, we proposed a novel technique in steganography within the digital images such is RGB, Gray Scale, Texture, Aerial images to achieve higher security, imperceptibility, capacity, and robustness as compared with existing methods.The experimental outcomes of the suggested approach prove further developed strength and justify the feasibility of our research.Through numerical simulations, we observed that the proposed strategy outperformed the nextbest current methodology by 5.561 percent in terms of PSNR Correlation score.Additionally, the proposed approach achieved a 6.43 percent better score in PSNR with a variable measure of code inserted in similar images with distinct dimensions.Furthermore, encrypting the same amount of information in images of varying sizes resulted in approximately 6.77 percent improvements.Embedding different sizes of a particular secret message in a different image (such as Gray, Texture, Aerial and RGB images) came out with about 5.466 percent of better score.
Revisiting Learnable Affines for Batch Norm in Few-Shot Transfer Learning

Batch normalization is a staple of computer vision models, including those employed in few-shot learning. Batch nor-malization layers in convolutional neural networks are composed of a normalization step, followed by a shift and scale of these normalized features applied via the per-channel trainable affine parameters <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\gamma$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\beta$</tex> . These affine param-eters were introduced to maintain the expressive powers of the model following normalization. While this hypothesis holds true for classification within the same domain, this work illustrates that these parameters are detrimen-tal to downstream performance on common few-shot trans-fer tasks. This effect is studied with multiple methods on well-known benchmarks such as few-shot classification on minilmageNet, cross-domain few-shot learning (CD-FSL) and META-DATASET. Experiments reveal consistent performance improvements on CNNs with affine unaccompanied batch normalization layers; particularly in large domain-shift few-shot transfer settings. As opposed to common practices in few-shot transfer learning where the affine pa-rameters are fixed during the adaptation phase, we show fine-tuning them can lead to improved performance.
Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach

One of the main focuses in distributed learning is communication efficiency, since model aggregation at each round of training can consist of millions to billions of parameters. Several model compression methods, such as gradient quantization and sparsification, have been proposed to improve the communication efficiency of model aggregation. However, the information-theoretic minimum communication cost for a given distortion of gradient estimators is still unknown. In this paper, we study the fundamental limit of communication cost of model aggregation in distributed learning from a rate-distortion perspective. By formulating the model aggregation as a vector Gaussian CEO problem, we derive the rate region bound and sum-rate-distortion function for the model aggregation problem, which reveals the minimum communication rate at a particular gradient distortion upper bound. We also analyze the communication cost at each iteration and total communication cost based on the sum-rate-distortion function with the gradient statistics of real-world datasets. It is found that the communication gain by exploiting the correlation between worker nodes is significant for SignSGD, and a high distortion of gradient estimator can achieve low total communication cost in gradient compression.
Deep Federated Learning-Based Privacy-Preserving Wind Power Forecasting

Given the growing installed capacity, wind energy will exert a profound impact on the flexibility of modern energy systems.Wind power forecasting is a practical solution for dealing with the attributed variations and uncertainties, balancing supply and demand, and improving the reliability of the system.To achieve more accurate and generalizable forecast models, comprehensive data sets, supplied by multiple wind farms owing to their spatio-temporal dependencies, are required.In addition, data aggregation/collaboration across many wind farms scattered around a country is difficult, if not impossible, due to complex administrative processes, industry competition, and data privacy and security concerns.This article offers federated learning-based wind energy forecasting as a novel decentralized collaborative modeling method capable of training a single model on data from many wind farms without jeopardizing the privacy or security of data.To this end, rather than sending private data across sites, local model parameters are securely transmitted.A comparison between the proposed private distributed model and non-private centralized and fully private localized models indicates the high performance of the proposed federated learning-based wind power forecasting with 87.96% accuracy.Enjoying the smoothing effect, the higher generalizability of the proposed model with 83.63% accuracy is also substantiated in comparison to localized and centralized approaches while the privacy of the underlying data is preserved.
OpenWolf: A Serverless Workflow Engine for Native Cloud-Edge Continuum

Nowadays, Serverless computing is emerging as one of the most used Cloud services. In particular, the Function as Service (FaaS) is bringing to Cloud consumers, developers, and devops many advantages in terms of service costs, speed of development, and ease of deployment. In fact, it stands to be a key technology for enabling the Cloud-Edge Continuum. Regardless of these features, it is still not possible to build FaaS native applications without a Cloud broker that coordinates the functions. Therefore, FaaS usage is limited to very simple and specific jobs. In this work, we brush up on the concept of Scientific Workflow using the FaaS paradigm, in order to realize full Native Serverless Workflows-based applications. We define a custom Workflow Manifest DSL used to describe function interactions, then we describe the implementation of an agent able to deploy architecture-independent functions and coordinate them according to the Manifest. Finally, federating the Cloud-Fog-Edge tiers in a single Continuum environment, we allow functions to take advantage of the Continuum tier’s characteristics where they are deployed. This project is called OpenWolf, it’s repository is published on GitHub, under GNU General Public License v3.0.
Fine-Grained Image Classification Based on Cross-Attention Network

Due to the high similarity of fine-grained image subclasses, small inter-class changes and large intra-class changes are caused, which leads to the difficulty of fine-grained image classification task. However, existing convolutional neural networks have been unable to effectively solve this problem. Aiming at the above-mentioned fine-grained image classification problem, this paper proposes a multi-scale and multi-level ViT model. First, through data augmentation techniques, the accuracy of fine-grained image classification can be effectively improved. Secondly, the small-scale input and large-scale input of the model make the input image have more feature ex-pressions. The subsequent multi-layeredness effectively utilizes the results of the previous layer of ViT, so that the data of the previous layer can be more effectively used in the next layer of ViT. Finally, cross-attention allows the results of two scale inputs to be fused in a reasonable way. The proposed model is competitive with current mainstream state-of-the-art methods on multiple datasets.
LIFT: Learning 4D LiDAR Image Fusion Transformer for 3D Object Detection

LiDAR and camera are two common sensors to collect data in time for 3D object detection under the autonomous driving context. Though the complementary information across sensors and time has great potential of benefiting 3D perception, taking full advantage of sequential cross-sensor data still remains challenging. In this paper, we propose a novel LiDAR Image Fusion Transformer (LIFT) to model the mutual interaction relationship of cross-sensor data over time. LIFT learns to align the input 4D sequential cross-sensor data to achieve multi-frame multi-modal information aggregation. To alleviate computational load, we project both point clouds and images into the bird-eye-view maps to compute sparse grid-wise self-attention. LIFT also benefits from a cross-sensor and cross-time data augmentation scheme. We evaluate the proposed approach on the challenging nuScenes and Waymo datasets, where our LIFT performs well over the state-of-the-art and strong baselines.