Reviewerly/exHarmony huggingface dataset:
Deep Appearance Consistent Human Pose Transfer

The fidelity of a pose transfer system depends on its ability to generate realistic images of a person under novel poses while preserving the desired human attributes (like face, hairstyle, and clothes). However, the visual fidelity is often compromised as the existing methods fail to extract rich appearance and pose features since they propagate the pose and the appearance information through the same pathway. Also, the repeated downsampling in these pathways leads to the loss of finer details, thus producing blurry results. Further, these methods use vanilla convolution that treats all the pixels as important and fail to focus primarily on significant regions needed for the desired transformation. This work proposes an appearance-consistent human pose transfer framework that progressively transforms the person in the source image to the desired target pose using the information from three pathways: an image pathway, a pose pathway, and an appearance pathway. We propose the use of gated convolution to dynamically extract features relevant for generating the transformed image. The appearance pathway generates an appearance code to produce an image consistent in appearance with that of the source image. We establish the efficacy of the proposed framework through an extensive set of experiments on DeepFashion, Market-1501, and the Action Class dataset. We also generate coherent action sequences through a given set of desired poses from the action class dataset that contains humans in three actions: golf, yoga/workouts, and tennis.
Contrastive JS: A Novel Scheme for Enhancing the Accuracy and Robustness of Deep Models

Deep learning technologies have been applied in various computer vision tasks in recent years. However, deep models suffer performance decay when some unforeseen data are contained in the testing dataset. Although data enhancement techniques can alleviate this dilemma, the diversity of real data is too tremendous to simulate. To tackle this challenge, we study a scheme for improving the robustness and efficiency of the deep network training process in visual tasks. Specifically, first, we build positive and negative sample pairs based on a class-sensitive strategy. Then, we construct a feature-consistent learning strategy based on contrastive learning to constrain the representations of interclass features while paying attention to the intraclass features. To extend the effect of the consistent strategy, we propose a novel contrastive Jensen-Shannon divergence consistency loss (JS loss) to restrict the probability distributions of different sample pairs. The proposed scheme successfully enhances the robustness and accuracy of the utilized model. We validated our approach by conducting extensive experiments in the domains of model robustness and few-shot object detection (FSOD). The results showed that the proposed method achieved remarkable gains over state-of-the-art (SOTA) methods. We obtained a 3.2% average improvement over the best-performing FSOD method.
Few-Shot Temporal Sentence Grounding via Memory-Guided Semantic Learning

Temporal sentence grounding (TSG) is an important yet challenging task in video-based information retrieval. Given an untrimmed video input, it requires the machine to predict the interested video segment semantically related to a given sentence query. Most existing TSG methods train well-designed deep networks to align the semantic between video-query pairs for activity grounding with a large amount of data. However, we argue that these works easily capture the selection biases of video-query pairs in a dataset rather than showing the robust reasoning abilities to handle the rarely appeared pairs (i.e., few-shot contents). To alleviate such limitation of the off-balance data distribution during the network training, in this paper, we propose a novel memory-augmented network called Memory-Guided Semantic Learning Network (MGSL-Net) to handle the few-shot TSG task for enhancing the model generalization ability. Specifically, given the matched video-query input, we first employ a graph attentive cross-modal interaction module to align their semantics in a cycle-consistent manner. Then, we develop the memory modules in both video and query domains to record the cross-modal shared semantic features in the domain-specific persistent memory. At last, a heterogeneous attention module is utilized to integrate the memory-enhanced multi-modal features in both video and query domains with further feature calibration. During training, the memory modules are dynamically associated with both common and rare cases to memorize all appeared contents, alleviating the issue of forgetting the few-shot contents. Therefore, in testing, the rare cases can be enhanced by retrieving the stored memories, improving the generalization ability of the model. Experimental results on three benchmarks (ActivityNet Caption, Charades-STA and TACoS) show the superiority of our method on both effectiveness and efficiency.
Error Detection Schemes Assessed on FPGA for Multipliers in Lattice-Based Key Encapsulation Mechanisms in Post-Quantum Cryptography

Advances in quantum computing have brought the need for developing public-key cryptosystems secure against attacks potentially enabled by quantum computers. In late 2017, the National Institute of Standards and Technology (NIST) launched a project to standardize one or more quantum computer-resistant public-key cryptographic algorithms. Among the main post-quantum algorithm classes, lattice-based cryptography is believed to be quantum-resistant. The standardization efforts including that of the NIST which will be concluded in 2022-2024 also affirm the importance of such algorithms. In this work, we propose error detection schemes for lattice-based key encapsulation mechanisms (KEMs). As our case study, we apply such schemes to the hardware accelerators for three post-quantum cryptographic algorithms that have advanced to the third round of the NIST PQC standardization process, i.e., FrodoKEM, Saber, and NTRU. The merit of the proposed schemes is that they can be applied to other applications and cryptographic algorithms that use multiplications in their hardware accelerators. The schemes proposed in this paper are based on recomputing with shifted, negated, and scaled operands. Moreover, we implement our fault detection schemes on field-programmable gate array (FPGA) family Kintex Ultrascale+ device xcku5p-sfvb784-1LV-i to benchmark the overheads induced and the performance degradation of the proposed approaches when added to the original architectures. The results show acceptable overhead and high error coverage for all three studied NIST PQC finalists.
"When the Code becomes a Crime Scene" Towards Dark Web Threat Intelligence with Software Quality Metrics

The increasing growth of illegal online activities in the so-called dark web—that is, the hidden collective of internet sites only accessible by a specialized web browsers—has challenged law enforcement agencies in recent years with sparse research efforts to help. For example, research has been devoted to supporting law enforcement by employing Natural Language Processing (NLP) to detect illegal activities on the dark web and build models for their classification. However, current approaches strongly rely upon the linguistic characteristics used to train the models, e.g., language semantics, which threatens their generalizability. To overcome this limitation, we tackle the problem of predicting illegal and criminal activities—a process defined as threat intelligence—on the dark web from a complementary perspective—that of dark web code maintenance and evolution— and propose a novel approach that uses software quality metrics and dark website appearance parameters instead of linguistic characteristics. We performed a preliminary empirical study on 10.367 web pages and collected more than 40 code metrics and website parameters using sonarqube. Results show an accuracy of up to 82% for predicting the three types of illegal activities (i.e., suspicious, normal, and unknown) and 66% for detecting 26 specific illegal activities, such as drugs or weapons trafficking. We deem our results can influence the current trends in detecting illegal activities on the dark web and put forward a completely novel research avenue toward dealing with this problem from a software maintenance and evolution perspective.
EFL-WP: Federated Learning-Based Workload Prediction in Inter-Cloud Environments

Resource allocation has been always a major concern of cloud providers. Workload prediction can effectively improve resource utility by providing information about resources available in the future. Machine learning-based workload prediction has been widely studied and deployed in large-scale clouds. However, workload prediction in Inter-Cloud environments has not been considered. Since more and more cloud services are orchestrated by containers (or virtual machines) distributed in multiple clouds, intelligent models for workload prediction should be trained based on trace data across clouds. Due to concerns raised by data privacy, the trace data of clouds may not be shared with each other, and general distributed training is not applicable. In this paper, we propose a general and flexible framework, namely EFL-WP, for training workload prediction models in the Inter-Cloud environment. EFL-WP adopts the federated learning approach and allows cloud providers to collaborate to train prediction models without sharing traces. EFL-WP considers the difference among workloads in different cloud (i.e., the Non-IID characteristic of traces) by using two novel techniques: participant selection mechanism and multi-global models aggregation. The former can prevent some local models trained on non-IID traces from participating in global aggregation, which is beneficial to the global model. The latter allows the coordinator to aggregate several global models according to the difference among local models. To further improve accuracy, EFL-WP adopts an ensemble inference strategy. Experimental results show that the proposed framework can be superior to baselines on both Alibaba dataset and Tencent Games Traces.
Lightweight Swarm Authentication

In this paper we describe a provably secure authentication protocol for resource limited devices. The proposed algorithm performs whole-network authentication using very few rounds and in a time logarithmic in the number of nodes. Compared to one-to-one node authentication and previous proposals, our protocol is more efficient: it requires less communication and computation and, in turn, lower energy consumption.
Learning to Answer Complex Visual Questions from Multi-View Analysis

Visual Question Answering (VQA) has received increasing attention in NLP research. Most VQA images focus on natural scenes. However, some images widely used in textbooks such as diagrams often contain complicated and abstract information (e.g. constructed graphs with logic and concepts). Therefore, Diagram Question answering (DQA) is a challenging but significant task, which is also helpful for machines to understand human cognitive behaviors and learning habits. On DQA task, we propose a multi-perspective understanding based visual question-answering method, which constructs a variety of different self-monitoring tasks in the form of prompts to help the model learn deeper information. For the first time, we propose a decoding method of “Cross Entropy constraint Decoding”, which can effectively constrain the content generated by the text when performing multiple selection tasks. This method has obtained SOTA in the evaluation task of CCKS-2022, which fully proves the effectiveness of the method.
Using Interleaving to Promote Inductive Learning in Educational Contexts

Abstract. Inductive learning, that is, abstracting conceptual knowledge, rules, or principles from exemplars, plays a major role in educational settings, from literacy acquisition to mathematics and science learning. Interleaving exemplars of different categories rather than presenting blocks might be a simple but powerful way to improve inductive learning by supporting discriminative contrast. Although a consistent advantage of interleaving has been demonstrated for visual materials, relatively few studies have examined educationally relevant materials, such as mathematical tasks, science problems, and verbal materials, and their results are mixed. We discuss how interleaving could be made fruitful for school learning of mathematics, science, and literacy acquisition. We conclude that interleaving should be tailored to the specific learning content and combined with supportive instructional measures that assist students in comparing exemplars for discriminating features. Finally, we sketch research gaps that revolve around the use of interleaved learning in the classroom.
Smart Translation for Physically Challenged People Using Machine Learning

In sign language, hand motions are one of the nonverbal communication modalities employed. It is most typically used by deaf and hard of hearing persons who have hearing or speech impairments. Difficulties communicating with one another or with regular people folks. Several sign language systems have been created by There are numerous producers all throughout the world, but they are neither versatile nor adaptable, end-user-friendly in terms of price. Our project's goal is to create a feasible system for communication for deaf people. This project is divided into two sections. (1) It translates an audio message into sign language, and (2) it translates images/video into text/speech. The first category we'll take input as audio, turns the audio recorded message into text message, and later displays predetermined Indian Sign Language(ISL) visuals or GIFs. The use of this technique facilitates communication between hearing and deaf persons. In the second category, we will gather photographs and train images with CNN and present the results.
A Complete Characterization of Security for Linicrypt Block Cipher Modes

We give characterizations of IND$-CPA security for a large, natural class of encryption schemes. Specifically, we consider encryption algorithms that invoke a block cipher and otherwise perform linear operations (e.g., XOR and multiplication by fixed field elements) on intermediate values. This class of algorithms corresponds to the Linicrypt model of Carmer & Rosulek (Crypto 2016). Our characterization for this class of encryption schemes is sound but not complete. We then focus on a smaller subclass of block cipher modes, which iterate over the blocks of the plaintext, repeatedly applying the same Linicrypt program. For these Linicrypt block cipher modes, we are able to give a sound and complete characterization of IND$-CPA security. Our characterization is linear-algebraic in nature and is easy to check for a candidate mode. Interestingly, we prove that a Linicrypt block cipher mode is secure if and only if it is secure against adversaries who choose all-zeroes plaintexts.
Dual Cross-Attention for Video Object Segmentation via Uncertainty Refinement

In this paper, we propose a novel approach to video object segmentation where dual streams consisting of a shared network and a special network are designed to constitute the feature memory of history frames. Cues of spatial position and time stamp are explicitly explored to learn the context for each frame in the video sequence. Self-attention and cross-attention are simultaneously exploited to extract more powerful features for segmentation. In contrast to STM and its variants, the proposed dual cross-attention performs in both appearance space and semantic space such that the derived features are more distinctive and then robust to similar overlapping objects. During decoding for segmentation, a local refinement technique is designed for the uncertain boundaries to obtain more precise and smooth object contours. Experimental results on the challenging benchmark datasets DAVIS-2016, DAVIS-2017, and YouTube-VOS demonstrate the effectiveness of our proposed approach to video object segmentation.
Virtual influencers in multimodal advertising

The article highlights new communication and advertising phenomenon – virtual influencers (VI) or virtual models employed by different brands to promote their collections and goods. Influencer marketing became an effective marketing strategy and profitable trend within the last twenty years. Influencers can be categorized in line with the content they generate, number of followers they have, their specialization, the impact they produce. Influencer marketing flourishes in social media. Celebrities who earned their fame offline are mega-influencers. Virtual influencers are transmedia 3D models created by high-caliber professionals via computer-generated imagery (CGI), animation, computer graphics, multimodality, copywriting with their bio, tastes, hobbies, lifestyle, even political preferences and social values. They are fond of social networking and have accounts in the most popular social media – Facebook, Twitter, Instagram, TikTok. Brands collaborate with social media influencers for narrowcasting young customers, mostly millennials who prefer social media to mass media and perceive virtual reality as a natural component of current civilization and everyday life. Experts believe virtual influencers’ rating is based on wow effect and they won’t be able to outperform celebrities despite high annual income as they are fictional and not emotional. Celebrities as mega-influencers and models fear virtual models will squeeze them out of marketing and advertising, other social media users have Uncanny Valley syndrome. Psychologists claim VI might trigger psychological problems for teenagers as they cannot attain glamorous lifestyle VI promote. Verbal behavior of virtual influencers is molded by copywriters and mirrors characteristics of English used in social media by teenagers and millennials – short simple sentences with acronyms visibly peppered with emoji and hashtags. Facebook, Instagram, TikTok, Twitter accounts of the most popular influencers Lil Miquela, Bermuda, Shudu were analyzed. The future of virtual influencers lies in synthesis of CGI and artificial intelligence (AI) to generate posts automatically and communicate with followers. Then virtual influencers can be used outside advertising and marketing to make impact on global audience
Target Search and Tracking with an Autonomous Quadcopter using Type-2 Fuzzy Logic Controller

In this study, an interval type-2 fuzzy logic controller has been proposed for the target search and tracking mission of an unmanned aerial vehicle. The visual data coming from the onboard camera sensor of the quadcopter have been processed to provide visual feedback from the environment and to detect the target. In the given mission, the quadcopter first performed a systematic search over the area until the target has been detected, and then proceeded with the tracking of the target. The ROS/Gazebo platform has been used to test the performance of the proposed control scheme. A fuzzy type-1 controller has been also developed to assess the performance of the type-2 controller. The results of the simulations indicate that type-2 fuzzy logic controller can provide better steady state and transient characteristics compared to its type-1 counterpart.
Deep Modality Independent Descriptor Learning for Optical and SAR Image Patch Matching

Due to the complementary information between multi-modal images, they are widely used in various applications. However, there are significant differences in appearance caused by different imaging mechanisms, which bring great challenges to multi-modal image patch matching. To solve this problem, this paper proposes a deep modality independent descriptor learning network (DMID-Net) for multi-modal image patch matching. DMID-Net computes the self-similarity of deep features as the structure descriptor for image patch matching, which is independent of image modality and shared between multi-modal images. Thus, the acquired deep modality independent descriptor(DMID) can reduce the influence of significant differences between multi-modal images, further improving the matching performances. Experimental results on a large number of optical and SAR image-pairs demonstrate the effectiveness of DMID-Net on multi-modal image patch matching.
Implicit White-Box Implementations: White-Boxing ARX Ciphers

Since the first white-box implementation of AES published twenty years ago, no significant progress has been made in the design of secure implementations against an attacker with full control of the device. Designing white-box implementations of existing block ciphers is a challenging problem, as all proposals have been broken. Only two white-box design strategies have been published this far: the CEJO framework, which can only be applied to ciphers with small S-boxes, and self-equivalence encodings, which were only applied to AES. In this work we propose implicit implementations, a new design of white-box implementations based on implicit functions, and we show that current generic attacks that break CEJO or self-equivalence implementations are not successful against implicit implementations. The generation and the security of implicit implementations are related to the self-equivalences of the non-linear layer of the cipher, and we propose a new method to obtain self-equivalences based on the CCZ-equivalence. We implemented this method and many other functionalities in a new open-source tool BoolCrypt, which we used to obtain for the first time affine, linear, and even quadratic self-equivalences of the permuted modular addition. Using the implicit framework and these self-equivalences, we describe for the first time a practical white-box implementation of a generic Addition-Rotation-XOR (ARX) cipher, and we provide an open-source tool to easily generate implicit implementations of ARX ciphers.
A combined-CNN model of TLS Traffic Recognition and Classification

The recognition and classification of network application protocol is the premise to implement network traffic engineering, security detection, access control, etc. However, the popularization of TLS (Transport Layer Security) marks the arrival of the era of traffic encryption, and the recognition and classification of network application protocol become a great challenge. This paper compares various existing technologies, analyzes their relevant era background and technical limitations in detail, and at last puts forward a combined-CNN model of TLS Traffic Recognition and Classification. The model generates a one-dimensional input from the session pcap files directly and generates a two-dimensional input by extracting the payload of IP packets related to the key negotiation aiming to achieve a context correlation ability. Finally, the two independent inputs are fused in the full connection layer. The experiment shows that the combined-CNN model performs well, especially in the precision of the recognition and classification of the network application protocol encrypted by TLS.