Reviewerly/exHarmony huggingface dataset:
Attention-Based Multi-Hypothesis Fusion for Speech Summarization

Speech summarization, which generates a text summary from speech, can be achieved by combining automatic speech recognition (ASR) and text summarization (TS). With this cascade approach, we can exploit state-of-the-art models and large training datasets for both subtasks, i.e., Transformer for ASR and Bidirectional Encoder Representations from Transformers (BERT) for TS. However, ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary. We investigate several schemes to combine ASR hypotheses. First, we propose using the sum of sub-word embedding vectors weighted by their posterior values provided by an ASR system as an input to a BERT-based TS system. Then, we introduce a more general scheme that uses an attention-based fusion module added to a pre-trained BERT module to align and combine several ASR hypotheses. Finally, we perform speech summarization experiments on the How2 dataset and a newly assembled TED-based dataset that we will release with this paper <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/nttcslab-sp-admin/TEDSummary. These experiments show that retraining the BERT-based TS system with these schemes can improve summarization performance and that the attention-based fusion module is particularly effective.
Unsupervised Domain-Adaptation-Based Tensor Feature Learning With Structure Preservation

Domain adaptation (DA) is widely used in computer vision and pattern recognition applications. It is an effective process where a model is trained on objects from the source domain to predict the categories of the objects in the target domain. The aim of feature extraction in domain adaptation is to learn the best representation of the data in a certain domain and use it in other domains. However, the main challenge here is the difference between the data distributions of the source and target domains. Also, in computer vision, the data are represented as tensor objects such as 3-D images and video sequences. Most of the existing methods in DA apply vectorization to the data, which leads to information loss due to failure to preserve the natural tensor structure in a low-dimensional space. Thus, in this article, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unsupervised DA-based tensor feature learning (UDA-TFL)</i> as a novel adapted feature extraction method that aims to avoid vectorization during transfer knowledge simultaneously; retain the structure of the tensor objects; reduce the data discrepancy between source and target domains; and represent the original tensor object in a lower dimensional space that is resistant to noise. Therefore, multilinear projections are determined to learn the tensor subspace without vectorizing the original tensor objects via an alternating optimization strategy. We integrate maximum mean discrepancy in the objective function to reduce the difference between source and target distributions. Extensive experiments are conducted on 39 cross-domain datasets from different fields, including images and videos. The promising results indicate that UDA-TFL significantly outperforms the state-of-the-art.
Towards Faster Functional Encryption for Privacy-preserving Machine Learning

Machine learning has become ubiquitous technology and is being used in many aspects of our daily lives. However, as machine learning works with a huge amount of data, many privacy concerns of users' data have arisen. The privacy issues are exacerbated when machine learning is performed in the cloud computing environment, i.e., machine learning as a service (MLaaS) settings. In recent years, the problem of privacy-preserving machine learning has been studied extensively, and different approaches have been proposed. Cryptography provides one possible solution, and several techniques based on homo-morphic encryption, secure multiparty computation, and newly growing functional encryption have been developed to address the privacy issues in machine learning. In this paper, we focus on privacy-preserving deep neural networks based on functional encryption. Most of the existing work based on functional encryption suffer from computational complexity and security issue. This paper proposes a novel methodology to compute the activation functions in a neural network in a secure and privacy-preserving manner using function-hiding inner product encryption (FHIPE). To our knowledge, this is the first work to focus on function-hiding inner product functional encryption for privacy-preserving machine learning. Also, we conduct experiments to speed up functional encryption in terms of computing the inner products. Our experimental results show 95x speedup for inner-product functional encryption-based secure activation and 10x speedup for FHIPE based secure activation function compared to earlier work in this field.
Evidential Combination of Classifiers for Imbalanced Data

It remains an important research topic for the classification of imbalanced data. There exist some methods to solve this problem, such as hybrid-sampling, over-sampling, and under-sampling. Each method has its own advantage, and different methods generally provide some complementary knowledge. We want to combine these three methods at the decision level in an appropriate way for achieving as good as possible classification performance. Evidence theory is expert at representing and combining uncertain information. So a new method called an evidential combination of classifiers (ECC) is proposed for dealing with imbalanced data. The classification result generated by different strategies (i.e., hybrid-sampling, over-sampling, or under-sampling) may have different reliabilities for query patterns. A cautious reliability evaluation rule is developed for each classification result based on the close neighborhoods. After that, the classification result is revised with a new belief redistribution way according to the reliability evaluation, and the probability/belief of one class can be partially transferred to other classes as well as the total ignorance, which is defined by the whole frame of classes. By doing this, we can reduce the error risk of each classification method. Then, the revised classification results from different methods are combined by evidence theory to make the final class decision. The effectiveness of the ECC method has been demonstrated using several experiments, and it shows that ECC can effectively improve the classification performance comparing with other related methods.
Defending against Membership Inference Attacks in Federated learning via Adversarial Example

Federated learning has attracted attention in recent years due to its native privacy-preserving features. However, it is still vulnerable to various membership inference attacks, such as backdoor, poisoning, and adversarial attacks. Membership Inference attack aims to discover the data used to train the model, which leads to privacy leaking ramifications on participants who use their local data to train the shared model. Recent research on countermeasure methods mainly focuses on protecting the parameters and has limitations in guaranteeing privacy while restraining the loss of the model. This paper proposes Fedefend, which applies adversarial examples to defend against membership inference attacks in federated learning. The proposed approach adds well-designed noise to the attack features of the target model of each iteration becomes an adversarial example. In addition, we also consider the utility loss of the model and use an adversarial method to generate noise to constrain the loss to a certain extent, which efficiently achieves a trade-off between privacy security and loss of the federated learning model. We evaluate the proposed Fedefend on two benchmark datasets, and the experimental results demonstrate that Fedefend has a good performance.
A Feature Fusion Based Deep Learning Model for Deepfake Video Detection

In the recent past, several tools developed for facial image manipulation utilizing techniques like FaceSwap and Deepfake with implausible success. Such tools enable to edit faces present in a video with a minimum effort and the outcomes are incredibly close to real faces. Despite their handful number of uses, they possess many unpleasant impacts like creating fake news and defaming celebrities on social media. Hence, pinpointing whether a video contains manipulated faces in it becomes a task of utmost importance. To this end, we have designed a face manipulation detection technique that can figure out whether a target video contains manipulated faces in it or not. To do this, we first crop out the target face from video frames. Next, we extract deep features using MesoInception from RGB as well as their YCbCr and HSV versions, which are then concatenated to form a single feature vector. On the top of this new feature vector, we add the classification layer to classify the images into fake and real. To evaluate our model performance, we use two publicly available video datasets, namely Celeb-DF (V2) and Faceforensis++. The experimental results lay out the effectiveness of the proposed model over some recently published models.
CBASH: Combined Backbone and Advanced Selection Heads With Object Semantic Proposals for Weakly Supervised Object Detection

Most recent object detection methods have achieved growing performance on public datasets. However, enormous efforts are needed for these methods due to the extensive annotations of ground-truth boxes. Weakly Supervised Object Detection (WSOD) methods hence have been proposed to solve this problem as only image-level annotations are required and then output bounding boxes related to the objects. In order to further elevate the weakly supervised detection methods on the extraction of reasonable features, the training of potential positive proposals, and the generation of proposals before training, we propose a new Combined Backbone and Advanced Selection Heads (CBASH) method with the proposals generated from the object semantic information. Specifically, Combined Backbone will make the unobvious object features more noticeable, Advanced Selection Heads promote more potential positive proposals to get training, and the generated object semantic proposals elevate the quality and quantity of positive proposals. The proposed method is evaluated on the challenging PASCAL VOC 2007 and 2012 benchmark datasets. Experimental results show that our proposed method can achieve improved performance on both VOC 2007 and VOC 2012 datasets and outperforms the existing state-of-the-art methods.
Characterization and Evaluation of IoT Protocols for Data Exfiltration

Data exfiltration relies primarily on network protocols for unauthorized data transfers from information systems. In addition to well-established Internet protocols (such as DNS, ICMP, or NTP, among others), adversaries can use newer protocols such as Internet of Things (IoT) protocols to inadvertently exfiltrate data. These IoT protocols are specifically designed to meet the limitations of IoT devices and networks, where minimal bandwidth usage and low power consumption are desirable. In this article, we review the suitability of IoT protocols for exfiltrating data. In particular, we focus on the Constrained Application Protocol (CoAP; version 1.0), the message queuing telemetry transport protocol (MQTT; in its versions 3.1.1 and 5.0), and the advanced message queuing protocol (AMQP; version 1.0). For each protocol, we review its specification and calculate the overhead and available space to exfiltrate data in each protocol message. In addition, we empirically measure the elapsed time to exfiltrate different amounts of data. In this regard, we develop a software tool (dubbed CHITON) to encapsulate and exfiltrate data within the IoT protocol messages. Our results show that both MQTT and AMQP outperform CoAP. Additionally, MQTT and AMQP protocols are best suited for exfiltrating data, as both are commonly used to connect to IoT cloud providers through IoT gateways and are therefore more likely to be allowed in business networks. Finally, we also provide suggestions and recommendations to detect data exfiltration in IoT protocols.
Visual Evaluation for Autonomous Driving

Autonomous driving technologies often use state-of-the-art artificial intelligence algorithms to understand the relationship between the vehicle and the external environment, to predict the changes of the environment, and then to plan and control the behaviors of the vehicle accordingly. The complexity of such technologies makes it challenging to evaluate the performance of autonomous driving systems and to find ways to improve them. The current approaches to evaluating such autonomous driving systems largely use a single score to indicate the overall performance of a system, but domain experts have difficulties in understanding how individual components or algorithms in an autonomous driving system may contribute to the score. To address this problem, we collaborate with domain experts on autonomous driving algorithms, and propose a visual evaluation method for autonomous driving. Our method considers the data generated in all components during the whole process of autonomous driving, including perception results, planning routes, prediction of obstacles, various controlling parameters, and evaluation of comfort. We develop a visual analytics workflow to integrate an evaluation mathematical model with adjustable parameters, support the evaluation of the system from the level of the overall performance to the level of detailed measures of individual components, and to show both evaluation scores and their contributing factors. Our implemented visual analytics system provides an overview evaluation score at the beginning and shows the animation of the dynamic change of the scores at each period. Experts can interactively explore the specific component at different time periods and identify related factors. With our method, domain experts not only learn about the performance of an autonomous driving system, but also identify and access the problematic parts of each component. Our visual evaluation system can be applied to the autonomous driving simulation system and used for various evaluation cases. The results of using our system in some simulation cases and the feedback from involved domain experts confirm the usefulness and efficiency of our method in helping people gain in-depth insight into autonomous driving systems.
Artificial Intelligence for the Design of Symmetric Cryptographic Primitives

This chapter provides a general overview of AI methods used to support the design of cryptographic primitives and protocols. After giving a brief introduction to the basic concepts underlying the field of cryptography, we review the most researched use cases concerning the use of AI techniques and models to design cryptographic primitives, focusing mainly on Boolean functions, S-boxes and pseudorandom number generators. We then point out two interesting directions for further research on the design of cryptographic primitives where AI methods could be applied in the future.
Improving the quality of predictive models in small data GSDOT: A new algorithm for generating synthetic data

In the age of the data deluge there are still many domains and applications restricted to the use of small datasets. The ability to harness these small datasets to solve problems through the use of supervised learning methods can have a significant impact in many important areas. The insufficient size of training data usually results in unsatisfactory performance of machine learning algorithms. The current research work aims to contribute to mitigate the small data problem through the creation of artificial instances, which are added to the training process. The proposed algorithm, Geometric Small Data Oversampling Technique, uses geometric regions around existing samples to generate new high quality instances. Experimental results show a significant improvement in accuracy when compared with the use of the initial small dataset as well as other popular artificial data generation techniques.
Practicing Moderation: Community Moderation as Reflective Practice

Many types of online communities rely on volunteer moderators to manage the community and maintain behavioral standards. While prior work has shown that community moderators often develop a deep understanding of the goals of their moderation context and sophisticated processes for managing disruptions,less is known about the processes through which moderators develop this knowledge. In this paper, we leverage Donald Schön's concept of reflective practice as a lens for exploring how community moderators develop the 'knowledge-in-action' that they use to perform their work. Drawing on interviews with 18 Twitch moderators, we conceptualize moderators as reflective practitioners, iteratively encountering novel situations and adjusting their practices and mental models. Our findings provide detailed insight into how community moderators reflect-in-action, re-evaluating in real-time their mental models of viewer intent and community goals, and reflect-on-action, conducting post hoc assessments of individual incidents and long-term changes to adjust their practice over time. Moderators working in teams reveal specific aspects of reflection facilitated by cooperative discussion, which we call 'groupwise reflective practice'. By identifying community moderation as a form of reflective practice, we can leverage insights gained from studying practitioners in other fields,providing theoretical and practical implications for the study and support of community moderation.
Mixed-Integer Motion Planning on German Roads Within the Apollo Driving Stack

Traffic situations with interacting participants pose difficulties for today’s autonomous vehicles to interpret situations and eventually achieve their own mission goal. Interactive planning approaches are promising solutions for solving such situations. However, most approaches are only assessed in simulation, as researchers lack the resources to operate an autonomous vehicle. Likewise, open-source stacks for autonomous driving, such as Apollo, provide competitive and resource-efficient state-of-the-art planning algorithms. However, promising planning concepts from research are usually not included within a reasonable time, possibly due to resource restrictions or technical limitations. Without evaluating these novel algorithms in reality, the benefits and shortcomings of proposed approaches cannot be thoroughly assessed. This work aims to contribute methodology and implementation to integrate a novel mixed-integer optimization-based planning algorithm in Apollo’s planning component and assess its performance and real-time capability in theory and practice. It discusses the necessary modifications of Apollo for deployment on a different vehicle and presents three real-world driving experiments on a public road alongside a detailed experience report. The driving experiments show a smooth trajectory tracking performance operating robustly under varying perception data quality and the real-time capability of the closed-loop system.
Diffusion-Based Graph Contrastive Learning for Recommendation with Implicit Feedback

Recent studies on self-supervised learning with graph-based recommendation models have achieved outstanding performance. They usually introduce auxiliary learning tasks that maximize the mutual information between representations of the original graph and its augmented views. However, most of these models adopt random dropout to construct the additional graph view, failing to differentiate the importance of edges. The insufficiency of these methods in capturing structural properties of the user-item interaction graph leads to suboptimal recommendation performance. In this paper, we propose a Graph Diffusion Contrastive Learning (GDCL) framework for recommendation to close this gap. Specifically, we perform graph diffusion on the user-item interaction graph. Then, the diffusion graph is encoded to preserve its heterogeneity by learning a dedicated representation for every type of relations. A symmetric contrastive learning objective is used to contrast local node representations of the diffusion graph with those of the user-item interaction graph for learning better user and item representations. Extensive experiments on real datasets demonstrate that GDCL consistently outperforms state-of-the-art recommendation methods.
Byzantine-Fault-Tolerant Consensus via Reinforcement Learning for Permissioned Blockchain-Empowered V2X Network

Blockchain has been forming the central piece of various types of vehicle-to-everything (V2X) network for trusted data exchange. Recently, permissioned blockchains garner particular attention thanks to their improved scalability and diverse needs from different organizations. One representative example of permissioned blockchain is Hyperledger Fabric (“Fabric”). Due to its unique execute-order procedure, there is a critical need for a client to select an optimal number of peers. The interesting problem that this paper targets to address is the tradeoff in the number of peers: a too large number will degrade scalability while a too small number will make the network vulnerable to faulty nodes. This optimization issue gets especially challenging in V2X networks due to mobility of nodes: a transaction must be executed, and the associated block must be committed before the vehicle leaves a network. To this end, this paper proposes a mechanism for selecting an optimal set of peers based on reinforcement learning (RL) to keep a Fabric-empowered V2X network impervious to dynamicity due to mobility. We model the RL as a contextual multi-armed bandit (MAB) problem. The results demonstrate the outperformance of the proposed scheme.
Bridging Video and Text: A Two-Step Polishing Transformer for Video Captioning

Video captioning is a joint task of computer vision and natural language processing, which aims to describe the video content using several natural language sentences. Nowadays, most methods cast this task as a mapping problem, which learns a mapping from visual features to natural language and generates captions directly from videos. However, the underlying challenge of video captioning, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> .e., sequence to sequence mapping across the different domains, is still not well handled. To address these problems, we introduce the polishing mechanism in an attempt to mimic human polishing process and propose a generate-and-polish framework for video captioning. In this paper, we propose a two-step transformer based polishing network (TSTPN) consisting of two sub-modules: the generation-module is to generate the caption candidate and the polishing-module is to gradually refine the generated candidate. Specifically, the candidate provides a global information of the visual contents in a semantically-meaningful order, where it is firstly considered as a semantic intersnubber to bridge the semantic gap between the text and video, with the cross-modal attention mechanism for better cross-modal modeling; and it secondly provides a global planning ability to maintain the semantic consistency and fluency of the whole sentence for better sequence mapping. In experiments, we present adequate evaluations to show that the proposed TSTPN achieves the comparable and even better performance than the state-of-the-art methods on the benchmark datasets.