Reviewerly/exHarmony huggingface dataset:
A Composite Network Model for Face Super-Resolution with Multi-Order Head Attention Facial Priors

Face super-resolution (FSR) aims to reconstruct high-resolution face images from low-resolution (LR) ones. Despite the progress made by deep convolutional neural networks (DCNNs) on FSR, convolutions struggle to relate spatially distant concepts and what is more, all image pixels and prior information (e.g., landmarks and facial component heatmaps) are treated equally regardless of importance, causing inaccuracy and decreasing the quality of face image recovery. To address these issues, in this paper we propose a composite network model for FSR with multi-order head attention facial priors. The proposed model contains a face hallucination transformer (FHT)-based network and a multi-order head attention (MOHA)-based DCNN. The FHT-based network can capture long-range dependencies and gradually increase resolution to achieve efficient and effective inference, while the MOHA-based DCNN exploits detailed and two-dimensional information of LR face images. Moreover, the novel generic submodule of the MOHA-based DCNN, namely Multi-Order Head Attention Network, can accurately model the relationship of facial components between spatial and channel dimensions. The proposed composite network model seamlessly integrates the advantages of DCNNs and transformers to super-resolve LR face images. When compared with state-of-the-art FSR methods on public benchmark datasets, the proposed model shows competitive recovery performance.
Enabling an autonomous agent sharing its minds, describing its conscious contents

We enable an autonomous agent (robot) to share its artificial mind, particularly its conscious content, with its audiences like humans or other agents. This effort is relying on the studies from cognitive modeling and supports the autonomous human robot interaction. LIDA is a biologically inspired systems-level cognitive model that explains and predicts how minds work. This model has been used in designing the controllers of intelligent autonomous agents. Here we propose to add a new description (sub) model into LIDA, allowing its agents to share their control structures, the artificial minds. We argue that the cognitive representations and processes introduced in LIDA may naturally serve as the source of the mind contents an agent can express out. We also proposed a type of description behavior based on LIDA’s action modules, which support the interactions between the agent and its real-world environments including human audiences. Through this description model, an agent’s mind will be more visible and accessible, so the agent’s intelligence can be assessed from not only its outside appearance and behaviors, but also its mind. In addition, the agent’s mind activities may associate with, so help explain certain behaviors it acted/acts. This helps the agent earn more understanding from its audiences so to being accepted by and engage to its living society better. Finally, as an agent’s controller, LIDA is designed heavily relying on the hypotheses of human minds, so its agent’s artificial mind contents are conceptually easy to understand to its human audiences.
Weakly supervised multi-class semantic video segmentation for road scenes

Weakly supervised multi-class video segmentation is one of the most challenging yet least studied research problems in computer vision. This study aims to investigate two main items: (1) effective feature update for temporal changes combined with feature reuse between temporal frames; and (2) learn object patterns in complex scenes specifically for videos under weak supervision. Associating image tags to visual appearance is not a straightforward learning task, especially for complex scenes. Therefore, in this paper, we present manifold augmentations to obtain reliable pixel labels from image tags. We propose a framework comprised of two key modules: a temporal split module for efficient video processing and a pseudo per-pixel seed generation module for precise pixel-level supervision. Particularly, in our model, we utilize and explore temporal correlations via temporal split module and temporal attention. To reuse the extracted features and incorporate temporal updates for precise and fast computation, a channel-wise temporal split mechanism between successive video frames is presented. Furthermore, we evaluated proposed modules in two additional settings: (1) fully or sparsely supervised road scene video segmentation; and (2) weakly supervised segmentation for complex road scene images. Experiments are conducted on the Cityscapes and CamVid datasets, using DeepLabv3 as segmentation network and LiteFlowNet to compute motion vectors.
Collaborative coded Distributed Compu-ting Scheme: A Two-phase Cooperative Game Approach

As a modern communication paradigm, Artificial intelligence based Internet of Things (AIoT) can provide an interactive platform across the globe to enrich the quality of networking services. With the AIoT paradigm, coded distributed computing (CDC) has recently emerged to be a promising solution to address the straggling effects in conventional distributed computing systems. In this article, we propose a novel CDC control scheme in the AIoT platform. Based on the cooperative game theory, the main challenges of our scheme are i) the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> value decision for the CDC process, and ii) edge node resource allocation for offloading tasks. Using the ideas of coalition game and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">weighted Nash social welfare solution (WNSWS)</i> , our proposed scheme is developed as a two-phase game model to achieve a mutually desirable solution. At the first phase, a dynamic coalition formation is proceeded to select the most adaptable edge nodes for the offloading subtasks. At the second phase, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">WNSWS</i> is adopted to effectively share each edge node’s computing resource. Based on the jointly design of these two cooperative games, we explore the synergy effect to optimize the CDC process. In the edge assisted distributed computing infrastructure, our reciprocal combinative approach can provide a fair-efficient solution through the sequential interactions of edge nodes and AIoT devices. In the performance evaluation, we provide extensive simulation analyses to show our scheme’s superiority by comparing with the existing baseline protocols.
LeakageVerif: Efficient and Scalable Formal Verification of Leakage in Symbolic Expressions

Side-channel attacks are a powerful class of attacks targeting cryptographic devices. Masking is a popular protection technique to thwart such attacks as it can be theoretically proven secure. However, correctly implementing masking schemes is a non-trivial task and error-prone. If several techniques have been proposed to formally verify masked implementations, they all come with limitations regarding expressiveness, scalability or accuracy. In this work, we propose a symbolic approach, based on a variant of the classical substitution method, for formally verifying arithmetic and boolean masked programs. This approach is more accurate and scalable than existing approaches thanks to a careful design and implementation of key heuristics, algorithms and data structures involved in the verification process. We present all the details of this approach and the open-source tool called <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LeakageVerif</monospace> which implements it as a python library, and which offers constructions for symbolic expressions and functions for their verification. We compare <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LeakageVerif</monospace> to three existing state-of-the-art tools on a set of 46 masked programs, and we show that it has very good scalability and accuracy results while providing all the necessary constructs for describing algorithmic to assembly masking schemes. Finally, we also provide the set of 46 benchmarks, named <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MaskedVerifBenchs</monospace> and written for comparing the different verification tools, in the hope that they will be useful to the community for future comparisons.
A Chinese Calligraphy-Writing Robotic System Based on Image-to-Action Translations and a Hypothesis Generation Net

This paper attempts to use a delta robot’s structure and reliable coordinates to develop a self-learning Chinese calligraphy-writing system that requires precise control. Ideally, to achieve human-like behavior, a delta robot can learn stroke trajectories autonomously and present the stroke beauty of calligraphy characters. Unfortunately, state-of-the-art approaches have not yet considered the presentation of stroke beauty resulting from angles of rotation and tilt of the brush. This paper presents an integrated system consisting of a stroke processing module, a hypothesis generation net (HGN) learning model, a delta robot, and an image capture module. Our approach utilizes both the stroke trajectories from the stroke processing module and angles information from the HGN learning model to automatically produce five degrees of freedom action instructions. Based on the instructions, the delta robot completes calligraphy writing. Then, the image capture module provides feedback to the writing system for error calculation and coordinate correction. We utilize the mean absolute percentage error to verify the performance of the writing results. A correction algorithm and linear regression were used to improve the error correction results (less than 2% error). After several cycles, the written results approached the target sample finally. Consequently, the written results produced by the delta robot prove that our proposed system is capable of self-learning and correction.
Static Laser Feedback Interferometry-Based Gaze Estimation for Wearable Glasses

Fast and robust gaze estimation is a key technology for wearable glasses, as it enables novel methods of user interaction as well as display enhancement applications, such as foveated rendering. State-of-the-art video-based systems lack a high update rate, integrateability, slippage robustness, and low power consumption. To overcome these limitations, we propose a model-based fusion algorithm to estimate gaze from multiple static laser feedback interferometry (LFI) sensors, which are capable of measuring distance toward the eye and the eye’s rotational velocity. The proposed system is ambient light robust and robust to glasses slippage. During evaluation, a gaze accuracy of 1.79° at an outstanding update rate of 1 kHz is achieved, while the sensors consume only a fraction of the power compared with the state-of-the-art video-based system.
Neural Predictor-Based Dynamic Surface Parallel Control for MIMO Uncertain Nonlinear Strict-Feedback Systems

We propose a neural predictor-based dynamic surface parallel control method for a class of uncertain nonlinear systems in this brief. The dynamics of the physical system is in the strict-feedback form and subject to multi-input, multi-output, and uncertain nonlinearities. The parallel control method is developed based on an ACP methodology, including three steps. Firstly, an artificial system to the physical system is developed using an echo state network-based neural predictor structure. Next, a high-order tuner-based computational experiment is developed to achieve online adaptive training of the echo state network. Finally, parallel execution is developed by using a second-order linear tracking differentiator-based dynamic surface control approach. The total closed-loop system can be proved to be input-to-state stable. The effectiveness of the proposed theoretical results is demonstrated by a simulation of trajectory tracking of an autonomous surface vehicle.
MASIC: Deep Mask Stereo Image Compression

Stereo image compression (SIC) aims to simultaneously compress a pair of left and right stereoscopic images, which can achieve higher compression efficiency than single image compression. In this paper, to benefit the SIC tasks, we collect a large real-world stereo image dataset, namely Palace, which is composed of hundreds of stereo image pairs at high-resolution. More importantly, we propose a novel mask stereo image compression network, namely MASIC, which can jointly compress the stereo images with high compression efficiency. Specifically, we first estimate the homography matrix between the stereo images through a regression model. Then, the left image is spatially transformed by the homography matrix, so that only the residual information needs to be encoded for the right image. To avoid the wrong guidance between stereo image pair, we propose a mask prediction module (MPM) to generate a multi-channel guided mask to navigate both the encoding and decoding processes. Based on the guided mask, we introduce a new mask conditional stereo entropy (MCSE) model, to fully explore the correlation between the stereo images in entropy coding. In the decoder, we develop a stereo decoding module to simultaneously decode the stereo images and enhance their compression quality. Experimental results show that our MASIC significantly advances the performance of SIC both quantitatively and qualitatively on a variety of datasets, and is robust to the change of parallax level between stereo images. The software codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/eecoder-dyf/MASIC</uri> .
Multiple POS Dependency-Aware Mixture of Experts for Frame Identification

Frame identification, which is finding the exact evoked frame for a target word in a given sentence, is a fundamental and crucial prerequisite for frame semantic parsing. It is generally seen as a classification task for target words, whose contextual representations are usually obtained using a neural network like BERT as an encoder, and enriched with a joint learning model or the knowledge of FrameNet. However, the distinction at a fine-grained level, such as the delicate differences in the information of syntax and PropBank roles caused by different parts-of-speech (POS) of target words, is neglected. We propose a Multiple POS Dependency-aware Mixture of Experts(MPDaMoE) network that integrates five types of information, consisting of the syntactic information of target words whose POS are nominal, adjectival, adverbial, or prepositional, and the PropBank role information of target words whose POS are only verbal. To better learn such information, a Mixture of Experts network is employed, in which every expert is a Graph Convolutional Network, to incorporate the different dependency information of target words. Our model outperforms state-of-the-art models in experiments on two benchmark datasets, which shows its effectiveness.
Video Frame Interpolation Based on Symmetric and Asymmetric Motions

Video frame interpolation is the task to synthesize intermediate frames between consecutive frames to increase the frame rate.Recently, various deep-learning techniques have been proposed to interpolate intermediate frames more reliably.However, many existing methods use either symmetric (linear) or asymmetric (non-linear) schemes only to estimate motions for the warping process, resulting in unreliable interpolation results.In this paper, we propose a novel video frame interpolation network based on both symmetric and asymmetric motion-based warping modules, which can deal with linear and non-linear motions, as well as occlusions, effectively.The symmetric warping module estimates symmetric motions to generate intermediate frames, while the asymmetric one predicts asymmetric motions to address non-linear motions and occlusion problems.We combine symmetric and asymmetric warping results to reconstruct intermediate frames more reliably.We also develop the frame synthesis network to refine the combined warping results.Experimental results demonstrate that the proposed network outperforms stateof-the-art video interpolation algorithms and that the two types of warping modules work effectively in a complementary manner on various benchmark datasets.
Swarm and Evolutionary Algorithms in Image Compression by F-Transform

This article investigates the application of swarm and evolutionary algorithms, namely the SOMA, DE, and GA, for optimizing the F-transform-based image compression.To do this, we introduce the cost function, evaluating the approximation of decompressed image to the original image, concerning the parameters that control the approximation quality of the F-transform.This function is then minimized by the selected algorithms to find optimal settings for image compression and decompression.We design experiments to compare the performance of the original F-transform method and the methods optimized by SOMA, DE, and GA on a dataset of 10 pictures.In all considered cases, the results obtained with the optimized method completely surpass those obtained by the original one.We also apply a statistical test (called Wilcoxon signed-rank test) for ranking the performance of selected algorithms in this issue.The results show that the SOMA and DE perform well in cases where the compressed image sizes are small.However, the GA algorithm shows outperformance in comparison with the others in more complicated cases where the compressed image size is bigger.The outperformance of the GA is in terms of decompression quality and computation time.Finally, we provide a visual comparison between the original F-transformbased method and the method optimized by the GA, tested on a 128 × 128 picture.The decompressed image by the latter is much sharper and more detailed than that obtained by the former.
ESA: External Space Attention Aggregation for Image-Text Retrieval

Due to the large gap between vision and language modalities, effective and efficient image-text retrieval is still an unsolved problem. Recent progress devotes to unilaterally pursuing retrieval accuracy by either entangled image-text interaction or large-scale vision-language pre-training in a brute force way. However, the former often leads to unacceptable retrieval time explosion when deploying on large-scale databases. The latter one heavily relies on the extra corpus to learn better alignment in the feature space while obscuring the contribution of the network architecture. In this work, we aim to investigate a trade-off to balance effectiveness and efficiency. To this end, on the premise of efficient retrieval, we propose the plug-and-play External Space attention Aggregation (ESA) module to enable element-wise fusion of modal features under spatial dimensional attention. Based on flexible spatial awareness, we further propose the Self-Expanding triplet Loss (SEL) to expand the representation space of samples and optimize the alignment of embedding space. The extensive experiments demonstrate the effectiveness of our method on two benchmark datasets. With the identical visual and textual backbones, our single model has outperformed the ensemble modal of similar methods, and our ensemble model can further expand the advantage. Meanwhile, compared with the vision-language pre-training embedding-base method that used 83× image-text pairs than ours, our approach not only surpasses in performance but also accelerates 3× on retrieval time. Codes and pre-trained models are available at https://github.com/KevinLight831/ESA.
An Algorithm for Finding Self-Orthogonal and Self-Dual Codes Over Gaussian and Eisenstein Integer Residue Rings Via Chinese Remainder Theorem

A code over Gaussian or Eisenstein integer residue ring is an additive group of vectors with entries in this integer residue ring which is closed under the action of constant multiplication by the Gaussian or Eisenstein integers. In this paper, we define the dual codes for the codes over the Gaussian and Eisenstein integer residue rings, and consider the construction of the self-dual codes. Because, in the Gaussian and Eisenstein integer rings, the uniqueness of the prime element decomposition holds in the same way as the one-variable polynomial rings over finite fields and the rational integer ring, we provide an efficient construction method for self-dual code generator matrices using that of moduli. As numerical examples, for Gaussian and Eisenstein integer rings, we enumerate and construct the self-dual codes for the actual moduli when the matrix size of the generator matrices is two.
Pareto Optimized Large Mask Approach for Efficient and Background Humanoid Shape Removal

We introduce deep learning-based methodology for removing unwanted human-like shapes in videos. The method uses Pareto-optimized Generative Adversarial Networks (GANs) technology, which is a novel contribution. The system automatically selects the Region of Interest (ROI) for each humanoid shape and uses a skeleton detection module to determine which humanoid shape to retain. The semantic masks of human like shapes are created using a semantic-aware occlusion-robust model that has four primary components: feature extraction, and local, global, and semantic branches. The global branch encodes occlusion-aware information to make the extracted features resistant to occlusion, while the local branch retrieves fine-grained local characteristics. A modified big mask inpainting approach is employed to eliminate a person from the image, leveraging Fast Fourier convolutions and utilizing polygonal chains and rectangles with unpredictable aspect ratios. The inpainter network takes the input image and the mask to create an output image excluding the background humanoid shapes. The generator uses an encoder-decoder structure with included skip connections to recover spatial information and dilated convolution and squeeze and excitation blocks to make the regions behind the humanoid shapes consistent with their surroundings. The discriminator avoids dissimilar structure at the patch scale, and the refiner network catches features around the boundaries of each background humanoid shape. The method is evaluated on two video object segmentation datasets (DAVIS and YouTube-VOS) and a database of 66 distinct video sequences of people behind a desk in an office environment. The efficiency was assessed using the Structural Similarity Index Measure (SSIM), Frechet Inception Distance (FID), and Learned Perceptual Image Patch Similarity (LPIPS) metrics and showed promising results in fully automated background person removal task.
A Scalable Small-Footprint Time-Space-Pipelined Architecture for Reservoir Computing

Reservoir computing (RC) is a lightweight machine learning algorithm for edge applications, which features a lower computation workload and one-time training process compared with the recurrent neural network (RNN) and Transformer. The existing RC accelerators still suffer considerable hardware overhead, which cannot satisfy the scalability on various edge FPGA/ASIC resource limits. This brief proposes a scalable small-footprint time-space-pipelined architecture for the cycle RC paradigm. The RC workload can be distributed onto a configurable number of processing elements (PE) under scalable resource limits. A time-space-pipelined PE is designed to efficiently execute the cycle RC computation, with the power-of-2 piecewise linearization circuits for non-linear activation function. Experimental results show that the proposed architecture can match a large variety of FPGA resource limits and ASIC power/area limits, with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$12.6\times $ </tex-math></inline-formula> energy efficiency improvement compared with the state-of-the-art RC accelerator.