Reviewerly/exHarmony huggingface dataset:
Fairness-aware Adversarial Perturbation Towards Bias Mitigation for Deployed Deep Models

Prioritizing fairness is of central importance in artificial intelligence (AI) systems, especially for those societal applications, e.g., hiring systems should recommend applicants equally from different demographic groups, and risk assessment systems must eliminate racism in criminal justice. Existing efforts towards the ethical development of AI systems have leveraged data science to mitigate biases in the training set or introduced fairness principles into the training process. For a deployed AI system, however, it may not allow for retraining or tuning in practice. By contrast, we propose a more flexible approach, i.e., fairness-aware adversarial perturbation (FAAP), which learns to perturb input data to blind deployed models on fairness-related features, e.g., gender and ethnicity. The key advantage is that FAAP does not modify deployed models in terms of param-eters and structures. To achieve this, we design a discriminator to distinguish fairness-related attributes based on latent representations from deployed models. Meanwhile, a perturbation generator is trained against the discriminator, such that no fairness-related features could be extracted from perturbed inputs. Exhaustive experimental evaluation demonstrates the effectiveness and superior performance of the proposed FAAP. In addition, FAAP is validated on real-world commercial deployments (inaccessible to model pa-rameters), which shows the transferability of FAAP, foreseeing the potential of black-box adaptation.
CNN-Transformer Based Generative Adversarial Network for Copy-Move Source/ Target Distinguishment

Copy-move forgery can be used for hiding certain objects or duplicating meaningful objects in images. Although copy-move forgery detection has been studied extensively in recent years, it is still a challenging task to distinguish between the source and the target regions in copy-move forgery images. In this paper, a convolutional neural network-transformer based generative adversarial network (CNN-T GAN) is proposed to distinguish the source and target regions in a copy-move forged image. A generator is first utilized to generate a mask that is similar to the groundtruth mask. Then, a discriminator is trained to discriminate the true image pairs from the false ones. When the discriminator cannot discriminate the true/false image pairs accurately, the generator can be used to obtain the final localization maps of copy-move forgery. In the generator, convolutional neural network (CNN) and transformer are exploited to extract the local features and global representations in copy-move forgery images, respectively. In addition, feature coupling layers are designed to integrate the features in CNN branch and transformer branch in an interactive way. Finally, a new Pearson correlation layer is introduced to match the similarity features in source and target regions, which can improve the performance of copy-move forgery localization, especially the localization performance on source regions. To the best of our knowledge, this is the first work to utilize transformer for feature extraction in copy-move forgery localization. The proposed method can not only detect the copy-move regions, but also distinguish the source and target regions. Extensive experimental results on several commonly used copy-move datasets have shown that the proposed method outperforms the state-of-the-art methods for copy-move detection.
Multimodal Fusion-Based Image Hiding Algorithm for Secure Healthcare System

The development of artificial intelligence plays a significant role of multimedia applications, especially in the healthcare domain. However, it has brought about the problem of sensitive information leakage. To address these challenges, an interesting multimodal fusion-based robust image hiding algorithm is proposed in this article. First, fused image is considered as mark image generated from MRI and CT images using nonsubsampled shearlet transform. Second, we employed principal component analysis to compute the appropriate coefficients of cover image for embedding purpose. Third, fused mark image is Arnold cat map encoded to address the security issue hidden mark media. Finally, the fusion of fractional dual-tree complex wavelet transform and randomized singular value decomposition is utilized to conceal encrypted fused mark media inside host image. Our findings show that the proposed algorithm outperforms some of the recent techniques in terms of high robustness and invisibility.
PCW-Net: Pyramid Combination and Warping Cost Volume for Stereo Matching

Existing deep learning based stereo matching methods either focus on achieving optimal performances on the target dataset while with poor generalization for other datasets or focus on handling the cross-domain generalization by suppressing the domain sensitive features which results in a significant sacrifice on the performance. To tackle these problems, we propose PCW-Net, a Pyramid Combination and Warping cost volume-based network to achieve good performance on both cross-domain generalization and stereo matching accuracy on various benchmarks. In particular, our PCW-Net is designed for two purposes. First, we construct combination volumes on the upper levels of the pyramid and develop a cost volume fusion module to integrate them for initial disparity estimation. Multi-scale receptive fields can be covered by fusing multi-scale combination volumes, thus, domain-invariant features can be extracted. Second, we construct the warping volume at the last level of the pyramid for disparity refinement. The proposed warping volume can narrow down the residue searching range from the initial disparity searching range to a fine-grained one, which can dramatically alleviate the difficulty of the network to find the correct residue in an unconstrained residue searching space. When training on synthetic datasets and generalizing to unseen real datasets, our method shows strong cross-domain generalization and outperforms existing state-of-the-arts with a large margin. After fine-tuning on the real datasets, our method ranks $$1^{st}$$ on KITTI 2012, $$2^{nd}$$ on KITTI 2015, and $$1^{st}$$ on the Argoverse among all published methods as of 7, March 2022.
On Generalizing Beyond Domains in Cross-Domain Continual Learning

Humans have the ability to accumulate knowledge of new tasks in varying conditions, but deep neural networks of-ten suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Many recent methods focus on preventing catastrophic forgetting under the assumption of train and test data following similar distributions. In this work, we consider a more realistic scenario of continual learning under domain shifts where the model must generalize its inference to an unseen domain. To this end, we encourage learning semantically meaningful features by equipping the classifier with class similarity metrics as learning parameters which are obtained through Mahalanobis similarity computations. Learning of the backbone representation along with these extra parameters is done seamlessly in an end-to-end manner. In addition, we propose an approach based on the exponential moving average of the parameters for better knowledge distillation. We demonstrate that, to a great extent, existing continual learning algorithms fail to handle the forgetting issue under multiple distributions, while our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
Sign language letters recognition model based on improved YOLOv5

To solve the problems of low recognition accuracy and poor robustness of existing sign language letters recognition models in scenes such as complex background interference and overlapping hands, in this paper, a YOLOv5-SLL sign language letters recognition model is proposed. Firstly, a Sign Language Letters (SLL) dataset contains a total of 3373 annotated sign language letters images is established. Secondly, the Convolutional Block Attention Module (CBAM) is introduced into the basis of the YOLOv5 network structure, so that the network can focus more on the extraction of hand features and reduces the interference of background noise. Finally, the redundant bounding boxes are optimized by using the soft-Non Maximum Suppression (soft-NMS) algorithm, which alleviates the problem of missed detection that easily occurs when the hands overlap. The experiments are conducted on the SLL dataset, and the results show that the proposed YOLOv5s-SLL model improves the mean Average Precision (mAP) by 2.03% compared with the YOLOv5 model, under the recognition speed is unchanged. And compared with other advanced algorithms such as SSD, YOLOv3, etc., the YOLOv5-SLL model still has a higher recognition accuracy and robustness.
A New Cross Ring Neural Network: Dynamic Investigations and Application to WBAN

Wireless body area network (WBAN) is a crucial tool in modern medical areas. This refers to the intelligent interconnection of wireless sensor nodes installed outside or inside the human body to recover some human vital signs. The security and bandwidth saturation remain crucial problems of this technology. In this work, a cross-ring-based neural network model (CRNN) is obtained from the Hopfield neural network definition and its complex dynamics are deeply analyzed. The steady-state study shows that the model has a unique equilibrium point and its analysis shows that the model’s dynamics is self-excited. Based on the two-parameter Lyapunov spectrum, the set of synaptic weights has been quickly identified to illustrate hyperchaotic behavior of the CRNN. With the help of bifurcation plots, graphs of the Lyapunov spectrum, and phase portraits, we have characterized periodic, chaotic, and hyperchaotic patterns in the model. Furthermore, an experimental setup has been built using microcontroller technology to further support the results of the numerical simulations. A parallel compressive sensing algorithm combined with a nonlinear congruent generator has been used to show the application of the newly designed CRNN to medical image compression and security. Security and compression performances indicate an efficient scheme with the capability to resist various attacks and the capability to produce low-size data image from large-size data image. For instant from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$256\times 256 $ </tex-math></inline-formula> image size, an encrypted and compressed output image is obtained at the compression rate of 0.5, processing time of 0.147 ms, encryption throughput of 1783.3 MBits/s, for a 2.9-GHz processor, the number of cycles to process the algorithm is 1.62. Consequently, the proposal can be applied to WBANs.
Risk-Aware Off-Road Navigation via a Learned Speed Distribution Map

Motion planning in off-road environments re-quires reasoning about both the geometry and semantics of the scene (e.g., a robot may be able to drive through soft bushes but not a fallen log). In many recent works, the world is classified into a finite number of semantic categories that often are not sufficient to capture the ability (i.e., the speed) with which a robot can traverse off-road terrain. Instead, this work proposes a new representation of traversability based exclusively on robot speed that can be learned from data, offers interpretability and intuitive tuning, and can be easily integrated with a variety of planning paradigms in the form of a costmap. Specifically, given a dataset of experienced trajectories, the proposed algorithm learns to predict a distribution of speeds the robot could achieve, conditioned on the environment semantics and commanded speed. The learned speed distribution map is converted into costmaps with a risk-aware cost term based on conditional value at risk (CVaR). Numerical simulations demonstrate that the proposed risk-aware planning algorithm leads to faster average time-to-goals compared to a method that only considers expected behavior, and the planner can be tuned for slightly slower, but less variable behavior. Furthermore, the approach is integrated into a full autonomy stack and demonstrated in a high-fidelity Unity environment and is shown to provide a 30% improvement in the success rate of navigation.
Performance Evaluation of Post-Quantum TLS 1.3 on Resource-Constrained Embedded Systems

Transport Layer Security (TLS) constitutes one of the most widely used protocols for securing Internet communications and has also found broad acceptance in the Internet of Things (IoT) domain. As we progress toward a security environment resistant to quantum computer attacks, TLS needs to be transformed to support post-quantum cryptography. However, post-quantum TLS is still not standardised, and its overall performance, especially in resource-constrained, IoT-capable, embedded devices, is not well understood. In this paper, we showcase how TLS 1.3 can be transformed into quantum-safe by modifying the TLS 1.3 architecture in order to accommodate the latest Post-Quantum Cryptography (PQC) algorithms from NIST PQC process. Furthermore, we evaluate the execution time, memory, and bandwidth requirements of this proposed post-quantum variant of TLS 1.3 (PQ TLS 1.3). This is facilitated by integrating the pqm4 and PQClean library implementations of almost all PQC algorithms selected for standardisation by the NIST PQC process, as well as the alternatives to be evaluated in a new round (Round 4). The proposed solution and evaluation focuses on the lower end of resource-constrained embedded devices. Thus, the evaluation is performed on the ARM Cortex-M4 embedded platform NUCLEO-F439ZI that provides 180 MHz clock rate, 2 MB Flash Memory, and 256 KB SRAM. To the authors’ knowledge, this is the first systematic, thorough, and complete timing, memory usage, and network traffic evaluation of PQ TLS 1.3 for all the NIST PQC process selections and upcoming candidate algorithms, that explicitly targets resource-constrained embedded systems.
Audio-Adaptive Activity Recognition Across Video Domains

This paper strives for activity recognition under domain shift, for example caused by change of scenery or camera viewpoint. The leading approaches reduce the shift in activity appearance by adversarial training and self-supervised learning. Different from these vision-focused works we leverage activity sounds for domain adaptation as they have less variance across domains and can reliably indicate which activities are not happening. We propose an audio-adaptive encoder and associated learning methods that discriminatively adjust the visual feature representation as well as addressing shifts in the semantic distribution. To further eliminate domain-specific features and include domain-invariant activity sounds for recognition, an audio-infused recognizer is proposed, which effectively models the cross-modal interaction across domains. We also introduce the new task of actor shift, with a corresponding audio-visual dataset, to challenge our method with situations where the activity appearance changes dramatically. Experiments on this dataset, EPIC-Kitchens and CharadesEgo show the effectiveness of our approach. Project page: https://xiaobai1217.github.io/DomainAdaptation.
Outpainting by Queries

Image outpainting, which is well studied with Convolution Neural Network (CNN) based framework, has recently drawn more attention in computer vision. However, CNNs rely on inherent inductive biases to achieve effective sample learning, which may degrade the performance ceiling. In this paper, motivated by the flexible self-attention mechanism with minimal inductive biases in transformer architecture, we reframe the generalised image outpainting problem as a patch-wise sequence-to-sequence autoregression problem, enabling query-based image outpainting. Specifically, we propose a novel hybrid vision-transformer-based encoder-decoder framework, named Query Outpainting TRansformer (QueryOTR), for extrapolating visual context all-side around a given image. Patch-wise mode's global modeling capacity allows us to extrapolate images from the attention mechanism's query standpoint. A novel Query Expansion Module (QEM) is designed to integrate information from the predicted queries based on the encoder's output, hence accelerating the convergence of the pure transformer even with a relatively small dataset. To further enhance connectivity between each patch, the proposed Patch Smoothing Module (PSM) re-allocates and averages the overlapped regions, thus providing seamless predicted images. We experimentally show that QueryOTR could generate visually appealing results smoothly and realistically against the state-of-the-art image outpainting approaches. Code is available at https://github.com/Kaiseem/QueryOTR .
DoFP+: An HTTP/3-Based Adaptive Bitrate Approach Using Retransmission Techniques

HTTP Adaptive Streaming (HAS) solutions use various adaptive bitrate (ABR) algorithms to select suitable video qualities with the objective of coping with the variations of network connections. HTTP has been evolving with various versions and provides more and more features. Most of the existing ABR algorithms do not significantly benefit from the HTTP development when they are merely supported by the most recent HTTP version. An open research question is “ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">How can new features of the recent HTTP versions be used to enhance the performance of HAS</i> ?” To address this question, in this paper, we introduce Days of Future Past+ (DoFP+ for short), a heuristic algorithm that takes advantage of the features of the latest HTTP version, HTTP/3, to provide high Quality of Experience (QoE) to the viewers. DoFP+ leverages HTTP/3 features, including (i) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stream multiplexing</i> , (ii) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stream priority</i> , and (iii) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">request cancellation</i> to upgrade low-quality segments in the player buffer while downloading the next segment. The qualities of those segments are selected based on an objective function and throughput constraints. The objective function takes into account two factors, namely the (i) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">average bitrate</i> and the (ii) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">video instability</i> of the considered set of segments. We also examine different strategies of download order for those segments to optimize the QoE in limited resources scenarios. The experimental results show an improvement in QoE by up to 33% while the number of stalls and stall duration for DoFP+ are reduced by 86% and 92%, respectively, compared to state-of-the-art ABR schemes. In addition, DoFP+ saves on average up to 16% downloaded data across all test videos. Also, we find that downloading segments sequentially brings more benefits for retransmissions than concurrent downloads; and lower-quality segments should be upgraded before other segments to gain more QoE improvement. Our source code has been published for reproducibility at https://github.com/cd-athena/ DoFP-Plus.
User Sentiment Analysis in Conversational Systems Based on Augmentation and Attention-based BiLSTM

Conversational Systems are increasingly substituting humans in many service industries. They aim to provide human-like interaction with users for task completion or chitchat in a conversation style. User sentiment analysis is an important task that can help better understand users’ behavior and satisfaction in conversations. Although some researchers have studied the problem of sentiment analysis, most of the existing methods are oriented toward general felds. To overcome the challenges of sentiment analysis, we propose a BE-Att-BiLSTM, which stands for an attention-based Bidirectional Long Short-Term Memory (BiLSTM) model. The proposed model uses pre-trained BERT, contextual embeddings and a combination of BiLSTM and attention mechanism for efficient sentiment analysis in conversations. In addition, text-augmentation techniques are leveraged to enhance the performance of the proposed model. Experimental results on a public benchmark dataset show an improved accuracy of 68.00% and an F1-score of 67.50%.
Universally Composable End-to-End Secure Messaging

We model and analyze the Signal end-to-end messaging protocol within the UC framework. In particular: Our modeling introduces additional innovations that enable arguing about the security of Signal irrespective of the underlying communication medium, as well as secure composition of dynamically generated modules that share state. These features, together with the basic modularity of the UC framework, will hopefully facilitate the use of both Signal-as-a-whole and its individual components within cryptographic applications. Two other features of our modeling are the treatment of fully adaptive corruptions, and making minimal use of random oracle abstractions. In particular, we show how to realize continuous key exchange in the plain model, while preserving security against adaptive corruptions.
Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis

Talking head synthesis is an emerging technology with wide applications in film dubbing, virtual avatars and online education. Recent NeRF-based methods generate more natural talking videos, as they better capture the 3D structural information of faces. However, a specific model needs to be trained for each identity with a large dataset. In this paper, we propose Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis, which can rapidly generalize to an unseen identity with few training data. Different from the existing NeRF-based methods which directly encode the 3D geometry and appearance of a specific person into the network, our DFRF conditions face radiance field on 2D appearance images to learn the face prior. Thus the facial radiance field can be flexibly adjusted to the new identity with few reference images. Additionally, for better modeling of the facial deformations, we propose a differentiable face warping module conditioned on audio signals to deform all reference images to the query space. Extensive experiments show that with only tens of seconds of training clip available, our proposed DFRF can synthesize natural and high-quality audio-driven talking head videos for novel identities with only 40k iterations. We highly recommend readers view our supplementary video for intuitive comparisons. Code is available in https://sstzal.github.io/DFRF/ .