Reviewerly/exHarmony huggingface dataset:
Can Multimodal Pointer Generator Transformers Produce Topically Relevant Summaries?

Due to the growth in demand for brief and pertinent multimedia material over the past few years, multimodal summarization has attracted a lot of study interest. Recently Transformers have been widely used for various sequence processing tasks due to their fast parallel processing ability compared to LSTMs. Although Multimodal Summarization (MS) has tractioned much research interest of late, a research gap exists in producing topic-relevant multimodal summaries. Since any summary deals with concise information, it should carry the essence of the topic from which it was derived. Further, due to the lack of alignment information among the images and the inter-modal segments, MS systems also face difficulty choosing appropriate pictorial summaries. To study these research questions, we propose a Multitask learning-based Multimodal Pointer Generator Transformer (MPGT), which utilizes the topic information of the samples to produce multimodal summaries. We also augment the popular MSMO dataset for this study with similar “On-Topic” and “Off-Topic” images. Our results show that inter-modal attention among images helps achieve better alignment in the visual modality and improves image precision scores. Our analysis also provides discussions on how we can further enhance topic-relevant MS systems.
Evaluating Robustness Against Adversarial Attacks: A Representational Similarity Analysis Approach

Deep Neural Network (DNN) provides state-of-the-art results in many application scenarios. However, the occasional instability of DNN is hard to be noticed under cover of its impressive prediction accuracy. As shown in adversarial attack, by making perturbations on original input, intentionally designed samples may make the model confused unexpectedly and produce misclassification. Existing methods are difficult to reveal this confusion cause in terms of DNN design structure and example inference flow. This work aims to interpret the robustness of DNN models on image classification tasks. By extracting Representation (which is the outputs of each layer), we evaluate and compare the robustness with the Representational Similarity Analysis (RSA) method. Specially, we compare the robustness of VGG, ResNet, and GoogLeNet against both the While-Box and Black-Box adversarial attacks on MNIST and CIFAR-10 datasets. Our experiments show that deep layers are vulnerable to adversarial attack, with the reason that small perturbation will be magnified as it goes deeper and deeper. We also find that relatively strong perturbation would lead to indiscriminative layers with high Representational Similarities, which indicates ineffective feature extracting, and as a result, most of the inputs of different classes would be classified into the same class. Finally, we demonstrate that the Representational Similarity could be quantitative metrics to evaluate the robustness of DNN models and their layers, which can help researchers better understand the layer-level reasons why adversarial attacks work.
Boundary-Aware Set Abstraction for 3D Object Detection

The basic components of a point-based 3D object detector is set abstraction (SA) layer, which down-samples points for better efficiency and enlarges receptive fields. However, existing SA layer only takes the relative locations among points into consideration, e.g. using furthest point sampling (FPS), while ignoring point features. Because the points on the objects take small proportion of space, the cascaded SA may miss to contain objects' points in the last layer, which will degrade the 3D object detection performances. We design a new lightweight and effective SA layer named Boundary-Aware Set Abstraction layer (BA-Net) to retain important foreground and boundary points during cascaded down-sampling. Technically, a lightweight point segmentation model (PSM) to compute the point-wise foreground scores is firstly embedded, then a Boundary Prediction Model (BPM) to detect the points on the object boundaries is proposed. These semantic scores are used to weight inter-node distances and the Boundary-aware Furthest Point down-Sampling (B-FPS) is conducted in this twisted distance space. Experimental results show that BA-Net can enhance the average accuracy (mAP) of the most competitive car class on the KITTI dataset by 1.14% and 1.06% in two different baseline. Code is available at https://github.com/HuangZhe885/Boundary-Aware-SA.
An Unseen Features Enhanced Text Classification Approach

In this paper, we discuss the issue of features that emerge during the prediction phase of a machine learning model, termed as unseen features. Because unseen features are absent from the vocabulary of the trained model, they are often rejected during the preprocessing stage of the learning model in standard machine learning approaches. We introduce the idea of unseen features and a method for identifying and using them for classification tasks. Because the dimension of feature vector required for trained machine learning model is going to differ upon incorporating unseen features of the testing data sample, it is not practical to directly incorporate unseen features since they only exist during the prediction phase of a machine learning model. As a result, the feature space for the training set is transformed to the embedding space which facilitates the use of unseen features. The proposed approach is empirically evaluated using standard metrics over three benchmark datasets in diverse circumstances (natural and balanced datasets) and on various text types – long-texts (aka structured texts) and short-texts (aka unstructured texts) considering five distinct classification algorithms. The experimental findings confirm the effectiveness of using unseen features during a machine learning model's deployment phase. The proposed unseen features enhanced technique outperforms the conventional approaches in both balanced class distribution and natural class distribution scenarios by a significant margin of at least 10%.
Light-weight Deep Extreme Multilabel Classification

Extreme multi-label (XML) classification refers to the task of supervised multi-label learning that involves a large number of labels. Hence, scalability of the classifier with increasing label dimension is an important consideration. In this paper, we develop a method called LightDXML which modifies the recently developed deep learning based XML framework by using label embeddings instead of feature embedding for negative sampling and iterating cyclically through three major phases: (1) proxy training of label embeddings (2) shortlisting of labels for negative sampling and (3) final classifier training using the negative samples. Consequently, LightDXML also removes the requirement of a re-ranker module, thereby, leading to further savings on time and memory requirements. The proposed method achieves the best of both worlds: while the training time, model size and prediction times are on par or better compared to the tree-based methods, it attains much better prediction accuracy that is on par with the deep learning based methods. Moreover, the proposed approach achieves the best tail-label prediction accuracy over most state-of-the-art XML methods on some of the large datasets.
Fruit Picker Activity Recognition with Wearable Sensors and Machine Learning

In this paper we present a novel application of detecting fruit picker activities based on time series data generated from wearable sensors. During harvesting, fruit pickers pick fruit into wearable bags and empty these bags into harvesting bins located in the orchard. Once full, these bins are quickly transported to a cooled pack house to improve the shelf life of picked fruits. For farmers and managers, the knowledge of when a picker bag is emptied is important for managing harvesting bins more effectively to minimise the time the picked fruit is left out in the heat (resulting in reduced shelf life). We propose a means to detect these bag-emptying events using human activity recognition with wearable sensors and machine learning methods. We develop a semi-supervised approach to labelling the data. A feature-based machine learning ensemble model and a deep recurrent convolutional neural network are developed and tested on a real-world dataset. When compared, the neural network achieves 86% detection accuracy.
BIT: Improving Image-text Sentiment Analysis via Learning Bidirectional Image-text Interaction

Exploring the interaction between image and text has a great strength for image-text sentiment analysis. However, most methods only focus on learning forward interaction in forward image-text features and fail to capture the backward interaction in backward image-text features, which leads to the loss of necessary information embedded in backward interaction. In this paper, Bidirectional Interaction Transformer (BIT) that models both forward and backward image-text interactions is proposed for image-text sentiment analysis. Specifically, we first encode image and text to forward and backward features. Then, these features are fed into Bidirectional Interaction Encoder (BIE) with Forward Interaction and Back Interaction branches to model bidirectional (i.e., forward and backward) image-text interaction. Finally, Two-scale Adaptive Gating Fusion (TAGF) is designed to adaptively fuse the forward and backward interactions learned by BIE. Extensive experiments conducted on two public datasets demonstrate the effectiveness of the proposed model.
Combination of Translation and Rotation in Dual Quaternion Space for Temporal Knowledge Graph Completion

Compared with static knowledge graphs (KGs) temporal KGs record the dynamic relations between entities over time, therefore, research on temporal Knowledge Graph Completion (KGC) attracts much attention. Temporal KGs exhibit complex temporal relation patterns, such as multiple relations. However, existing methods can hardly model all the relation patterns and apply to the temporal KGs. In this paper, we propose a novel temporal KGC method that Combining Translation and Rotation (ComTR) in Dual Quaternion Space for temporal KGC. Specifically, we use dual-quaternion-based multiplication to model timestamps and relations as the combination of translation and rotation operations. We analyze the relation patterns of temporal KGs in detail and demonstrate that our method can model all the relation patterns in temporal KGs. Empirically, we show that ComTR can achieve the state-of-the-art performances over four temporal KGC benchmarks datasets.
Vision Graph Convolutional Network for Writer-Independent Offline Signature Verification

As a biometric feature, handwritten signatures have various applications in finance, law, and business. The existing signature verification methods are mostly based on convolutional neural networks or Transformer based models. In this paper, we aim to implement writer independent offline handwritten signature verification by proposing an end-to-end method, Signature Verification Graph Convolutional Network (SigGCN). In SigGCN, signature images are first transformed into graphstructured data with additional position embedding to retain spatial information. The reason for this is that we expect graphstructured data to perform better in the capture of complicated relationships than CNN-based and Transformer based networks. We then use a multi-layer graph convolutional network to aggregate node information while updating the graph information. After obtaining the graph representation of signatures, efficient model training is performed using our proposed margin-based focal loss function to calculate the loss based on the Euclidean distance of two signatures. We conduct experiments on the CEDAR, BHSig260-Bengali, and BHSig260-Hindi datasets, and results obtained show that the proposed approach achieves remarkable performance and demonstrates great potential in solving real-world verification problems.
Camera Coach: Activity Recognition and Assessment Using Thermal and RGB Videos

Human Activity Recognition (HAR) has sparked a lot of attention because of its wide range of applications in sports, animation, simulation, and entertainment. There are many applications based on HAR, like healthcare monitoring, entertainment applications, and athletes' performance analysis. During the Corona pandemic, people took remote training as a safe and easy solution to do sports activities during the closure period, so there was a need to use smart systems to help people to train alone. In this paper, we expand our work on our dataset Multi-Modal Dataset of Sports (MMDOS); this dataset contains a variety of data forms from different sensors, including RGB videos, inertial motion data, depth, and thermal data, all synchronized with regard to the activities/exercises performed. The dataset consists of four workout exercises: free squats, shoulder press, push-ups, and lunges. The exercises are performed by 50 participants indoors at a fitness center. Professional trainers label the data with the type of activity and the participant's mistakes. Data are collected from different sensors, including Red-Green-Blue (RGB) videos, depth videos and skeleton data, inertial motion units (IMU) data, and thermal data; all are synchronized. From these modalities, we specifically used RGB, thermal videos, and 3-D skeleton of depth videos to recognize and asses the four activities. We built a smart Gym coaching model using classical machine learning (ML) and deep learning (DL) methods. The proposed system achieved an outstanding performance of 99% for activity recognition and an error of 0.75 out of 10 for activity assessment.
FedRRA: Reputation-Aware Robust Federated Learning against Poisoning Attacks

As an emerging machine learning paradigm, federated learning (FL) allows multiple participants to train a shared global model collaboratively on decentralized data while protecting data privacy. But traditional FL is susceptible to adversarial poisoning attacks, the global model in an FL system poisoned by adversaries may fail to converge or present accuracy degradation. To defend against data poisoning and model poisoning attacks simultaneously, we propose a Federated learning framework with a Reputation-aware Robust Aggregation (FedRRA) rule. It involves a two-step adversary detection: 1) a DBSCAN algorithm excludes models with obviously biased parameters and 2) an accuracy evaluation process punishes models with low accuracy. The reputation calculated in the two-step detection determines that clients with low reputations are removed before the aggregation, which alleviates the negative influence of models corrupted by adversaries. Extensive experiments demonstrate that FedRRA is superior to the state-of-the-art robust FL baseline in defending against model poisoning and data poisoning attacks.
A Multi-modal Multi-task based Approach for Movie Recommendation

An online recommendation system is one of the desires of digital e-commerce sectors and the OTT platforms like Amazon Prime, Netflix, SonyLiv, etc. In recent times, with an increase in the interaction of users with the different e-commerce platforms and then analyzing their liking-disliking essence, the recommendation system tries to predict the preference of the user for recommending new items that may capture his attention. In the current study, a multi-task-based architecture is designed to solve the multi-modal movie recommendation problem. Here our hypothesis is that solving two related tasks, namely (a) genre classification of movies and (b) rating identification for a user-movie pair, helps in generating good quality movie embeddings in an end-to-end setting without using a rating vector. For generating the representation of movies, unlike the state-of-the-art techniques, feature vectors extracted from multiple modalities like textual summary, audio and video information present in the movie trailers, and meta-data information are fused together. For representing the user, average representations of movies that are liked by the user are considered. Different multitasking models, fully shared (FS), shared-private (SP), and adversarial shared-private (ASP) feature models are developed for solving the above-mentioned two tasks simultaneously, genre classification, and user-movie rating prediction. For experimental purposes, MMTF-14K: a multifaceted movie trailer feature dataset was extended by incorporating textual features and meta-data information, and a multi-modal version of the MovieLens-100K dataset is used. Results of different multitasking models are shown in terms of RMSE and different rank-based metrics. The proposed multi-task model along with the adversarial training outperforms the state-of-the-art models when applied to the MMTF-14K and multi-modal version of MovieLens-100K datasets.
Chapter: Exploiting Convolutional Neural Network Adapters for Self-Supervised Speech Models

Self-supervised learning (SSL) is a powerful technique for learning representations from unlabeled data. Transformer based models such as HuBERT, which consists of a feature extractor and transformer layers, are leading the field in the speech domain. SSL models are fine-tuned on a wide range of downstream tasks, which involves re-training the majority of the model for each task. Previous studies have introduced applying adapters, which are small lightweight modules commonly used in Natural Language Processing (NLP) to adapt pre-trained models to new tasks. However, such efficient tuning techniques only provide adaptation at the transformer layer, but failed to perform adaptation at the feature extractor. In this paper, we propose CHAPTER, an efficient tuning method specifically designed for SSL speech model, by applying CNN adapters at the feature extractor. Using this method, we can only fine-tune fewer than 5% of parameters per task compared to fully fine-tuning and achieve better and more stable performance. We empirically found that adding CNN adapters to the feature extractor can help the adaptation on emotion and speaker tasks. For instance, the accuracy of SID is improved from 87.71 to 91.56, and the accuracy of ER is improved by 5%. Such finding suggests that when fine-tuning a SSL speech model, we should adapt different parameter-efficient tuning techniques depending on the downstream task.
Fast All-day 3D Object Detection Based on Multi-sensor Fusion

Realtime 3D object detection in all-day conditions is a challenging task for autonomous vehicles. Various image and point cloud based object detection methods have been proposed. Image based detectors are sensitive to illumination changes and cannot get accurate 3D information. Point cloud based detectors are less convenient for acceleration and deployment on commonly used hardware due to the unstructured nature of point cloud data, making it prohibitive for mobile platforms with limited computing resources in large-scale outdoor scenes. Frustum based 3D object detector first detects 2D objects in the image and then extracts frustum point cloud according to the cross-calibration parameters. Small-scale frustum point clouds can be used for 3D object detection, hence substantially accelerating the computation. However, when objects are missed in the first stage image based detector, the whole algorithm will fail to detect them. In this paper, we extended the frustum based 3D object detector by leveraging more sensor modalities. Our method combines two frustum based 3D object detecting branches in which visible light image and thermal image are used for 2D ROI extracting respectively. After obtaining 3D object proposals from the two branches, 3D non-maximum suppression is conducted to get the final detections. Experiments tested on our experimental autonomous vehicle show that our proposed method is capable of detecting 3D objects fast in various complex environments.
Explainable AI via Linguistic Summarization of Black Box Computer Vision Models

There is an ever-growing demand to characterize and understand AI as it is integrated into everyday life. Linguistic summaries have been previously used to provide natural language descriptions of data and models. However, the number of possible summaries increases rapidly with the number of data attributes. To make sense of the vast number of possible linguistic statements for a system, we introduce a hierarchical approach for generating and ranking linguistic statements. Each description of the model is assigned a value based on user criteria, allowing summaries to be tailored to specific users. We provide visualizations of the generation of summaries for a popular computer vision detector on a synthetically generated dataset.
Security Risk and Attacks in AI: A Survey of Security and Privacy

This survey paper provides an overview of the current state of AI attacks and risks for AI security and privacy as artificial intelligence becomes more prevalent in various applications and services. The risks associated with AI attacks and security breaches are becoming increasingly apparent and cause many financial and social losses. This paper will categorize the different types of attacks on AI models, including adversarial attacks, model inversion attacks, poisoning attacks, data poisoning attacks, data extraction attacks, and membership inference attacks. The paper also emphasizes the importance of developing secure and robust AI models to ensure the privacy and security of sensitive data. Through a systematic literature review, this survey paper comprehensively analyzes the current state of AI attacks and risks for AI security and privacy and detection techniques.
Learning Collaborative Multi-Target Search for A Visual Drone Swarm

In this paper, we propose a multi-agent reinforcement learning approach, POCA-Mix, to achieve collaborative multi-target search with a visual drone swarm. The proposed approach leverages the benefits of curriculum learning and mixed credit assignment to guide the drone swarm in per-forming the search task with only local visual perception in a constrained 3D environment. To validate the performance of the proposed approach, we conducted simulation experiments with various combinations regarding the number of drones and targets. The results demonstrate that the proposed approach outperforms other baseline methods such as PPO and MA-POCA with a higher success rate in different scenarios.
Collaborative Global-Local Networks for Fine-Grained Few-shot Learning

In this paper, we focus on addressing the fine-grained few-shot learning task, which aims at recognizing the novel classes with a few fine-grained samples for each class, via learning a transfer model with some sample-rich base classes. Compared with the few-shot learning tasks on the coarse-grained classes, the fine-grained few-shot learning task is more challenging due to the intra-class dissimilarity and the inter-class similarity. To this end, we propose a Collaborative Global-Local Network (CGLNet) to exploit both global and local visual features for few-shot fine-grained classification. Through the features processing and multiple distance functions, the proposed CGLNet could effectively exploit the global and local information in features, thus it brings in benefits for the downstream few-shot tasks. To evaluate the effectiveness of the proposed approach, we conduct extensive experiments on three popular fine-grained datasets. The experimental results show that our model achieves superior performance to state-of-the-art methods.