Reviewerly/exHarmony huggingface dataset:
Hindering Adversarial Attacks with Multiple Encrypted Patch Embeddings

In this paper, we propose a new key-based defense focusing on both efficiency and robustness. Although the previous key-based defense seems effective in defending against adversarial examples, carefully designed adaptive attacks can bypass the previous defense, and it is difficult to train the previous defense on large datasets like ImageNet. We build upon the previous defense with two major improvements: (1) efficient training and (2) optional randomization. The proposed defense utilizes one or more secret patch embeddings and classifier heads with a pre-trained isotropic network. When more than one secret embeddings are used, the proposed defense enables randomization on inference. Experiments were carried out on the ImageNet dataset, and the proposed defense was evaluated against an arsenal of state-of-the-art attacks, including adaptive ones. The results show that the proposed defense achieves a high robust accuracy and a comparable clean accuracy compared to the previous key-based defense.
Distributed Intelligent Resource Orchestration

The evolution to microservices and cloud-native technologies represents a transformative milestone in the telecommunications sector, offering the potential to unlock the capabilities of 5G/6G advancements. New cloud-based applications can be built from the ground up as containerized microservices, breaking a complex application into a series of smaller specialized and manageable services [1]. Efficient resource orchestration in cloud-native infrastructure is a critical challenge, ensuring uninterrupted service delivery. Kubernetes has emerged as de facto standard orchestration platform. However, its client-server architecture and storage on disk introduced latency which is not suitable for real-time communications. To address this gap, we propose DIRO (Distributed Intelligent Resource Orchestration), an orchestration platform for low latency microservices. Our proposal focus on a new scheduling architecture approach leverages in-memory data grid and distributed resource predictions to reduce scheduling delay.
COREL: Constrained Reinforcement Learning for Video Streaming ABR Algorithm Design Over mmWave 5G

The adaptive bitrate selection (ABR) mechanism, which decides the bitrate for each video chunk is an important part of video streaming. There has been significant interest in developing Reinforcement-Learning (RL) based ABR algorithms because of their ability to learn efficient bitrate actions based on past data and their demonstrated improvements over wired, 3G and 4G networks. However, the Quality of Experience (QoE), especially video stall time, of state-of-the-art ABR algorithms including the RL-based approaches falls short of expectations over commercial mmWave 5G networks, due to widely and wildly fluctuating throughput. These algorithms find optimal policies for a multi-objective unconstrained problem where the policies inherently depend on the predefined weight parameters of the multiple objectives (e.g., bitrate maximization, stall-time minimization). Our empirical evaluation suggests that such a policy cannot adequately adapt to the high variations of 5G throughput, resulting in long stall times. To address these issues, we formulate the ABR selection problem as a constrained Markov Decision Process where the objective is to maximize the QoE subject to a stall-time constraint. The strength of this formulation is that it helps mitigate the stall time while maintaining high bitrates. We propose COREL, a primal-dual actor-critic RL algorithm, which incorporates an additional critic network to estimate stall time compared to existing RL-based approaches and can tune the optimal dual variable or weight to guide the policy towards minimizing stall time. Our experiment results across various commercial mmWave 5G traces reveal that COREL reduces the average stall time by a factor of 4 and the 95th percentile by a factor of 2.
Persona-based Dialogue Generation with Sentence Embedding for Prolonged Human-Robot Communication

It is one of the most fundamental and challenging problems for current dialogue generation systems to maintain the consistency of dialogue logic during the conversation. Also, the lack of an open-source annotated persona-based dialogue dataset may lead to insufficient training volume for the model. Besides, the computational time and computational memory required by the attention mechanism have become drastically large. In order to overcome these problems, in this work, we propose a vertical-structure model based on the BERT model with the sentence embedding method. The model generates a raw response based on the sentence embeddings of context and persona and finally revises the raw response according to the persona. Moreover, an understanding task is designed for the BERT decoder to have a better revision ability. Considering the difference between the generation and the understanding models, three kinds of input methods are designed for each part of the model. Comparative and experimental results are presented using publicly available datasets.
Old Damaged Photo Recovery with Style Transfer-Based Data Augmentation

Restoring damaged old photos to their original state can enhance the viewing and comprehension of the historical context and time period captured by these photos. Recent studies attempted to restore these photos using proposed methods, but the results often suffer from blurring and loss of details. Therefore, we propose a restoration model designed explicitly for damaged old photos in this paper. Furthermore, due to the limited quantity of damaged old photos compared to other task datasets, we employ data augmentation techniques to increase the quantity and diversity of training data for the restoration method. The experimental results demonstrate that our restoration method surpasses existing methods across various evaluation metrics and provides a satisfying viewing experience.
A UAV Indoor Obstacle Avoidance System Based on Deep Reinforcement Learning

This paper presents a vision-based unmanned aerial vehicle (UAV) indoor obstacle avoidance using a deep reinforcement learning (DRL). The system consists of two parts a depth map compression and a UAV control. For the depth map compression part, the pre-trained Variational AutoEncoder model is used to improve obstacle avoidance and reduce the training time. The states include the UAV information, gray images and compressed features. A dueling double deep recurrent Q network model is used to control the UAV. The network is trained in the AirSim simulation. To validate the performance of the proposed algorithm some simulations are conducted. The results show that the proposed algorithm can avoid obstacles at a fast speed in a narrow space, and fly through a difficult L-shaped corner in an indoor simulation.
Multiple-Backup Resource Allocation for Virtual Machines with Probabilistic Protection in Cloud

The paper proposes a model to minimize backup capacity while limiting the probability of unsuccessful primary resource backup during multiple physical machine (PM) failures. In this model, when a PM containing virtual machines (VMs) fails, VMs in that PM can be recovered by available backup resources in backup PMs. By setting the priority of protecting VMs and backup PMs that each VM is protected by, we obtain the probability of unsuccessful backup. We introduce heuristic approaches to allocate backup capacity with the least required capacity. The proposed model can reduce the total required backup capacity compared to the conventional models.
Enhanced Residual Fourier Transformation Network for Lightweight Image Super-resolution

Residual blocks have been widely used in the single image super-resolution (SR), boosting the performance of SR results. The stacking of residual blocks helps to grab more information needed for the recovery of high-resolution (HR) images, but at the cost of larger model size and computational complexity. To address this problem, we propose a lightweight enhanced residual fourier transformation network (ERFTN). To ensure the network is lightweight enough while being capable of capturing both spatial and frequency information, we introduce a novel enhanced residual fourier transformation block (ERFTB) with an enhanced spatial attention (ESA) block. In addition, we adopt the contrastive loss for training acceleration without additional parameters. Extensive experiments demonstrate that our method can achieve SR performance superior to the state-of-the-art SR methods while reducing approximately half of the number of parameters (e.g., with 190K number of parameters, achieving 32.39dB in PSNR on Urban100 dataset).
Method of Tracking Moving Objects Combining Kalman Filter and K-Means

The goal of this study is to obtain particles position and velocity information by Kalman filter and locate animals by using the K … means method for the particles that are close to each other. Finally the animals can be regarded as a single object The velocity of a target is observed using a spatio-temporal differential method that introduces a compensation method. In the compensation method, how to determine the compensation amount is important. Determining the compensation amount based on the prediction of Kalman filter, it is possible to reduce the error included in the particles information. A large number of particles are generated on a frame image, and their positions and velocities are predicted using a Kalman filter. The states of the particles are then estimated based on the observed velocity using the spatio-temporal differential method, These process is performed for each frame and the particles are used to track the moving objects.
Efficient Incremental Text-to-Speech on GPUs

Incremental text-to-speech, also known as streaming TTS, has been increasingly applied to online speech applications that require ultra-low response latency to provide an optimal user experience. However, most of the existing speech synthesis pipelines deployed on GPU are still non-incremental, which uncovers limitations in high-concurrency scenarios, especially when the pipeline is built with end-to-end neural network models. To address this issue, we present a highly efficient approach to perform real-time incremental TTS on GPUs with Instant Request Pooling and Module-wise Dynamic Batching. Experimental results demonstrate that the proposed method is capable of producing high-quality speech with a first-chunk latency lower than 80ms under 100 QPS on a single NVIDIA A10 GPU and significantly outperforms the non-incremental twin in both concurrency and latency. Our work reveals the effectiveness of high-performance incremental TTS on GPUs.
CapFormer: A Space-Time Video Description Model using Joint-Attention Transformer

Transformers in video understanding are becoming popular due to the recent success of vision transformers. However, video transformers are still emerging and require various tricks to understand video tasks, such as action detection, classification, and description. The need for such tricks is due to videos’ temporal and spatial space, which requires different techniques to understand the context better. The most critical component is the attention block, which gives different results if tackled differently. In video description techniques, the model designs are mainly complex due to the integration needed between visual and language contexts. In this work, we proposed a simple yet efficient video description model that relies entirely on an attention mechanism. The model uses a joint-attention mechanism to learn the spatial and temporal context of video frames with their description context. Surprisingly, this integration setting suggests a more straightforward way to achieve the same task achieved by complex networks, and thus, it is efficient in terms of training. To validate the design, we evaluated the proposed architecture on a large video description dataset (MSR-VTT) and compared its result with various works, and it showed promising results over other designs in terms of the ROUGE metric.
StyleStegan: Leak-free Style Transfer Based on Feature Steganography

In modern social networks, existing style transfer methods suffer from a serious content leakage issue, which hampers the ability to achieve serial and reversible stylization, thereby hindering the further propagation of stylized images in social networks. To address this problem, we propose a leak-free style transfer method based on feature steganography. Our method consists of two main components: a style transfer method that accomplishes artistic stylization on the original image and an image steganography method that embeds content feature secrets on the stylized image. The main contributions of our work are as follows: 1) We identify and explain the phenomenon of content leakage and its underlying causes, which arise from content inconsistencies between the original image and its subsequent stylized image. 2) We design a neural flow model for achieving loss-free and biased-free style transfer. 3) We introduce steganography to hide content feature information on the stylized image and control the subsequent usage rights. 4) We conduct comprehensive experimental validation using publicly available datasets MS-COCO and Wikiart. The results demonstrate that StyleStegan successfully mitigates the content leakage issue in serial and reversible style transfer tasks. The SSIM performance metrics for these tasks are 14.98% and 7.28% higher, respectively, compared to a suboptimal baseline model.
Investigation of Ensemble of Self-Supervised Models for Speech Emotion Recognition

Traditional deep learning based speech emotion recognition (SER) methods are restricted to the lack of emotional speeches. Recently, the self-supervised learning (SSL) speech representation model is emerging and has become the state-of-the-art method for SER because of its ability to learn a more generalized emotion representation. Some SSL models adopt a similar network architecture, but utilize various training strategies or pre-trained speech data to learn speech representations from different perspectives, thus containing complementary information for the SER task. In this paper, we conduct an investigation of ensemble learning with six SSL models for SER using three kinds of ensemble methods: decision ensemble, late feature ensemble, and early feature ensemble. The six SSL models adopted in this paper are Wav2vec 2.0 Base, HuBERT Base, WavLM Base, WavLM Base+, UniSpeech-SAT Base, and UniSpeech-SAT Base+. Our experimental results on three datasets show that: 1) all three kinds of ensemble learning methods can significantly improve the SER performance, demonstrating that there may exist a large amount of complementary information between different SSL models; 2) in terms of the decision ensemble methods, the evidence theory (ET) based fusion performs better than the average-based fusion because of its ability to deal with the uncertainty of different models; 3) the ET-based decision ensemble is the best in comparative ensemble methods because of its superior performance and lower computational cost.
Feature Augmentation Reconstruction Network for Few-Shot Image Classification

In recent years, deep learning has achieved significant success in various computer vision tasks, notably in image classification. Nonetheless, labeled data is frequently insufficient in real-world applications, presenting significant impediments to the efficacy of deep learning models. This issue is especially pronounced in few-shot image classification tasks. To alleviate the above problem, we propose to augment features in few-shot image classification to increase sample diversity. This is achieved through learning and fitting the features' distribution, generating pseudo-data based on the distribution, and ultimately augmenting the features. Specifically, the adherence of the sample to the Gaussian distribution is learned using both local and global information, and then feature data is generated based on the distribution for feature augmentation. We evaluated the proposed method on four benchmark datasets, and the experimental results show that our method achieves state-of-the-art performance compared to prevailing few-shot image classification methods in most experimental settings.
Exploring a CLIP-Enhanced Automated Approach for Video Description Generation

Visual storytelling is a learned ability that humans have developed through the course of their evolution. In contrast to written records and descriptions, people are now sharing glimpses of their lives through short videos. Human conversations largely rely on visual and auditory inputs, followed by corresponding feedbacks. For machines, the conversion of images to texts serves as a bridge between visual and linguistic information, enabling machine-human interactions more naturally. Video captioning—involving automatically generating textual descriptions from videos—is one of the core technologies that enable such applications. In this work, we present CLIP-CAP, an automatic method for transforming visual contents to concise textual descriptions. We investigate the CLIP pretraining model as well as its potential in this task. Through experiments on the ActivityNet Captions dataset, we show that the proposed CLIP-CAP model outperforms existing video captioning methods in terms of several different metrics.
Feature Selection Based on Clonal Selection Algorithm for Image Steganalysis

With the rapid development of steganography, researches on the adaptive steganography have made steganalysis more and more difficult. In order to improve the detection accuracy, there are plenty of works introducing complex high-dimensional features to steganalysis, among which the rich model steganalysis features are particularly classical. However, there are many redundant features in high-dimensional features, which greatly increase the time and computational cost. How to eliminate redundant features while maintaining detection accuracy is still a research hotspot. In this paper, we propose a novel feature selection method based on Clonal Selection Algorithm (FS-CSA) for image steganalysis, which separates features into multiple populations according to the idea of niche technique, and proposes a feature similarity function to calculate the affinity of features in each population, and finally uses FS-CSA to select features based on the affinity in each population in turn. The experimental results show that FS-CSA can significantly reduce the feature dimensionality while maintaining the detection accuracy, where the Spatial Rich Model (SRM) features can be reduced to about 8000 dimensions, and the Gabor Filter Residual (GFR) features can be reduced to about 7000 dimensions. The crossover experiments show that the steganalysis features selected by FS-CSA are also generalizable to any steganography algorithm.
Implicit Neural Representation for Video Coding Through Progressive Feature Extraction

Implicit neural representation for video (INRV) utilizes a neural network to represent a video without relying on an explicit pixel-wise representation of RGB values, and, therefore an INRV-based video coding method aims to compress a neural network and transmit its parameters through a bit-stream. In this paper, we propose a novel rate-distortion (R-D) optimized INRV-based video coding method through a progressive feature extraction module (PFEM). The PFEM consists of a series of residual blocks (RBs) that can progressively control the quality of the reconstruction, by searching the optimal network architecture. By increasing the number of RBs, the quality of a reconstruction frame is improved. However, the quality comes at the expense of overheads in transmission. In the proposed method, the number of the RBs are decided to provide the best trade-off between the frame quality and bits. Our decision model is based upon the R-D theory. Experimental results demonstrated that the proposed method provides a superior coding gain to the conventional video coding standards.
Implementation and Analysis on Backpropagating Refinement Scheme for Interactive Image Segmentation

BRS is the first CNN-based interactive image segmentation algorithm to refine segmentation results based on a backpropagation scheme. In this paper, we give a detailed description of how BRS operates and demonstrate how to implement the algorithm. In BRS, user-provided clicks are first converted into interaction maps, which are then concatenated with the RGB image and provided as input to a segmentation network. In the test phase, performing the forward pass in the network generates an initial segmentation map. However, the user-annotated pixels may be mislabeled in the initial result. BRS refines this result by correcting the mislabeled pixels. We implement this BRS algorithm in PyTorch and publish the source codes. Moreover, we first show that BRS can reach the perfect IoU ratio of 1.0 in most cases and delineate objects more accurately than a variant of BRS, called f-BRS.
Sparsity Exploration for Structured and Unstructured Weight Formations in CNN Architecture

The Convolutional Neural Network (CNN) is a frequently used algorithm used technique for processing pixel data and visual images. Nevertheless, the utilization of extensive network sizes poses significant obstacles to the underlying technology’s throughput and energy efficiency. Various methods have been devised to enhance the efficiency and computing speed of Convolutional Neural Network (CNN) acceleration, improving the accuracy of the CNN process. Compression techniques have been employed in the algorithmic aspect to minimize the number of parameters in each layer, enhancing the detection process’s efficiency. Pruning is a compression technique that involves the removal of neurons and synapses that do not play a substantial role in the detection process, hence minimizing their impact on the Neural Network layer. The compression method generates sparse data, wherein a significant amount of data has a value of zero. Various pruning techniques provide distinct types of sparse data, encompassing organized structured sparse and unstructured sparse data.The consequent form of sparse data structure has many advantages and downsides. Based on an analysis of various extant sparse data structures, an innovative accelerator has been developed to perform convolution operations using various types of sparse data obtained via distinct pruning procedures. The design architecture outlined in this study involves the removal of all zero-valued data weights through a pruning technique following the training process. The accelerator will process all non-zero weights by performing the convolution process promptly, utilizing the "weight stream" data. The findings from the conducted design testing indicate that the utilization of trimmed sparse data leads to improved performance in terms of faster output during the convolution process. The researchers conducted experiments utilizing diverse types of sparse weighted data, encompassing both structured and unstructured forms. These experiments provided evidence that the hardware accelerator is capable of performing convolutional computing operations with different variations of sparse data. The convolution procedure utilizing the developed accelerator is projected to yield a reduction of 56% in the computation results of all layers in YoloV3-Tiny. In the context of data calculations, including a sparsity percentage of up to 30%, it can be observed that the convolution process exhibits a reduction of 25% in relation to the overall convolution process.