Reviewerly/exHarmony huggingface dataset:
Invertible Image Decolorization

Invertible image decolorization is a useful color compression technique to reduce the cost in multimedia systems. Invertible decolorization aims to synthesize faithful grayscales from color images, which can be fully restored to the original color version. In this paper, we propose a novel color compression method to produce invertible grayscale images using invertible neural networks (INNs). Our key idea is to separate the color information from color images, and encode the color information into a set of Gaussian distributed latent variables via INNs. By this means, we force the color information lost in grayscale generation to be independent of the input color image. Therefore, the original color version can be efficiently recovered by randomly re-sampling a new set of Gaussian distributed variables, together with the synthetic grayscale, through the reverse mapping of INNs. To effectively learn the invertible grayscale, we introduce the wavelet transformation into a UNet-like INN architecture, and further present a quantization embedding to prevent the information omission in format conversion, which improves the generalizability of the framework in real-world scenarios. Extensive experiments on three widely used benchmarks demonstrate that the proposed method achieves a state-of-the-art performance in terms of both qualitative and quantitative results, which shows its superiority in multimedia communication and storage systems.
A Hyperbolic-to-Hyperbolic Graph Convolutional Network

Hyperbolic graph convolutional networks (GCNs) demonstrate powerful representation ability to model graphs with hierarchical structure. Existing hyperbolic GCNs resort to tangent spaces to realize graph convolution on hyperbolic manifolds, which is inferior because tangent space is only a local approximation of a manifold. In this paper, we propose a hyperbolic-to-hyperbolic graph convolutional network (H2H-GCN) that directly works on hyperbolic manifolds. Specifically, we developed a manifold-preserving graph convolution that consists of a hyperbolic feature transformation and a hyperbolic neighborhood aggregation. The hyperbolic feature transformation works as linear transformation on hyperbolic manifolds. It ensures the transformed node representations still lie on the hyperbolic manifold by imposing the orthogonal constraint on the transformation sub-matrix. The hyperbolic neighborhood aggregation updates each node representation via the Einstein midpoint. The H2H-GCN avoids the distortion caused by tangent space approximations and keeps the global hyperbolic structure. Extensive experiments show that the H2H-GCN achieves substantial improvements on the link prediction, node classification, and graph classification tasks.
Improving OCR-based Image Captioning by Incorporating Geometrical Relationship

OCR-based image captioning aims to automatically describe images based on all the visual entities (both visual objects and scene text) in images. Compared with conventional image captioning, the reasoning of scene text is required for OCR-based image captioning since the generated descriptions often contain multiple OCR tokens. Existing methods attempt to achieve this goal via encoding the OCR tokens with rich visual and semantic representations. However, strong correlations between OCR tokens may not be established with such limited representations. In this paper, we propose to enhance the connections between OCR tokens from the viewpoint of exploiting the geometrical relationship. We comprehensively consider the height, width, distance, IoU and orientation relations between the OCR tokens for constructing the geometrical relationship. To integrate the learned relation as well as the visual and semantic representations into a unified framework, a Long Short-Term Memory plus Relation-aware pointer network (LSTM-R) architecture is presented in this paper. Under the guidance of the geometrical relationship between OCR tokens, our LSTM-R capitalizes on a newly-devised relation-aware pointer network to select OCR tokens from the scene text for OCR-based image captioning. Extensive experiments demonstrate the effectiveness of our LSTM-R. More remarkably, LSTM-R achieves state-of-the-art performance on TextCaps, with the CIDEr-D score being increased from 98.0% to 109.3%.
ArabicDialects: An Efficient Framework for Arabic Dialects Opinion Mining on Twitter Using Optimized Deep Neural Networks

The rapid development of tools for communication such as social networks, tweeting and Whatsapp has generated a large mass of important textual data. Also, the COVID-19 pandemic has inflamed social networks, hence the automatic analysis of opinions has become paramount. The purpose of this paper is to analyze Arabic tweets in terms of positivity, negativity, or neutrality.In analyzing the opinions of the Arabic language, a real challenge is encountered, which lies in the use of different dialects (Egyptian, Saudian, Maghrebian, Gulfian, Levantine, Syrian ⋯). In this paper, we introduce two major components: The first employs six machine learning (ML) methods, including Decision Trees (DT), Logistic Regression (LR), k Nearest Neighbors (K-NN), Random Forests (RF), Support Vector Machines (SVM), and Nave Bayes (NB), with the TF-IDF method acting as the feature extraction.While, the second part consists of testing three variants of Deep Learning (DL) based on multiplicative Long Short Term Memory (mLSTM), Long Short Term Memory (LSTM), and Gated Recurrent Unit (GRU) by applying word embedding as the input vector. The experimental study was validated using three Arabic language corpora (TEAD, ATSAD, and ASTD) and two learning modes (Hold out and 10-folds cross validation). The obtained results in terms of Accuracy (ACC), Precesion (PREC), Recall (REC), and F1-score (F1) show a clear performance for DL techniques based on a 10-folds strategy compared to the state-of-the-art. The experiments shown in the paper reveal that the proposed DL models accomplished the best results.
Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation

Semantic segmentation is a crucial image understanding task, where each pixel of image is categorized into a corresponding label. Since the pixel-wise labeling for ground-truth is tedious and labor intensive, in practical applications, many works exploit the synthetic images to train the model for real-word image semantic segmentation, i.e., Synthetic-to-Real Semantic Segmentation (SRSS). However, Deep Convolutional Neural Networks (CNNs) trained on the source synthetic data may not generalize well to the target real-world data. To address this problem, there has been rapidly growing interest in Domain Adaption technique to mitigate the domain mismatch between the synthetic and real-world images. Besides, Domain Generalization technique is another solution to handle SRSS. In contrast to Domain Adaption, Domain Generalization seeks to address SRSS without accessing any data of the target domain during training. In this work, we propose two simple yet effective texture randomization mechanisms, Global Texture Randomization (GTR) and Local Texture Randomization (LTR), for Domain Generalization based SRSS. GTR is proposed to randomize the texture of source images into diverse unreal texture styles. It aims to alleviate the reliance of the network on texture while promoting the learning of the domain-invariant cues. In addition, we find the texture difference is not always occurred in entire image and may only appear in some local areas. Therefore, we further propose a LTR mechanism to generate diverse local regions for partially stylizing the source images. Finally, we implement a regularization of Consistency between GTR and LTR (CGL) aiming to harmonize the two proposed mechanisms during training. Extensive experiments on five publicly available datasets (i.e., GTA5, SYNTHIA, Cityscapes, BDDS and Mapillary) with various SRSS settings (i.e., GTA5/SYNTHIA to Cityscapes/BDDS/Mapillary) demonstrate that the proposed method is superior to the state-of-the-art methods for domain generalization based SRSS.
Overview of the Neural Network Compression and Representation (NNR) Standard

Neural Network Coding and Representation (NNR) is the first international standard for efficient compression of neural networks (NNs). The standard is designed as a toolbox of compression methods, which can be used to create coding pipelines. It can be either used as an independent coding framework (with its own bitstream format) or together with external neural network formats and frameworks. For providing the highest degree of flexibility, the network compression methods operate per parameter tensor in order to always ensure proper decoding, even if no structure information is provided. The NNR standard contains compression-efficient quantization and deep context-adaptive binary arithmetic coding (DeepCABAC) as core encoding and decoding technologies, as well as neural network parameter pre-processing methods like sparsification, pruning, low-rank decomposition, unification, local scaling and batch norm folding. NNR achieves a compression efficiency of more than 97&#x0025; for transparent coding cases, i.e. without degrading classification quality, such as top-1 or top-5 accuracies. This paper provides an overview of the technical features and characteristics of NNR.
An Overview and Analysis of Hybrid Encryption: The Combination of Symmetric Encryption and Asymmetric Encryption

In the current scenario, various forms of information are spread everywhere, especially through the Internet. A lot of valuable information is contained in the dissemination, so security issues have always attracted attention. With the emergence of cryptographic algorithms, information security has been further improved. Generally, cryptography encryption is divided into symmetric encryption and asymmetric encryption. Although symmetric encryption has a very fast computation speed and is beneficial to encrypt a large amount of data, the security is not as high as asymmetric encryption. The same pair of keys used in symmetric algorithms leads to security threats. Thus, if the key can be protected, the security could be improved. Using an asymmetric algorithm to protect the key and encrypting the message with a symmetric algorithm would be a good choice. This paper will review security issues in the information transmission and the method of hybrid encryption algorithms that will be widely used in the future. Also, the various characteristics of algorithms in different systems and some typical cases of hybrid encryption will be reviewed and analyzed to showcase the reinforcement by combining algorithms. Hybrid encryption algorithms will improve the security of the transmission without causing more other problems. Additionally, the way how the encryption algorithms combine to strength the security will be discussed with the aid of an example.
DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering

Video question answering is a challenging task, which requires agents to be able to understand rich video contents and perform spatial-temporal reasoning. However, existing graph-based methods fail to perform multi-step reasoning well, neglecting two properties of VideoQA: (1) Even for the same video, different questions may require different amount of video clips or objects to infer the answer with relational reasoning; (2) During reasoning, appearance and motion features have complicated interdependence which are correlated and complementary to each other. Based on these observations, we propose a Dual-Visual Graph Reasoning Unit (DualVGR) which reasons over videos in an end-to-end fashion. The first contribution of our DualVGR is the design of an explainable Query Punishment Module, which can filter out irrelevant visual features through multiple cycles of reasoning. The second contribution is the proposed Video-based Multi-view Graph Attention Network, which captures the relations between appearance and motion features. Our DualVGR network achieves state-of-the-art performance on the benchmark MSVD-QA and SVQA datasets, and demonstrates competitive results on benchmark MSRVTT-QA datasets. Our code is available at https://github.com/MMIR/DualVGR-VideoQA.
eHealthChain—a blockchain-based personal health information management system

Medical IoT devices that use miniature sensors to collect patient's bio-signals and connected medical applications are playing a crucial role in providing pervasive and personalized healthcare. This technological improvement has also created opportunities for the better management of personal health information. The Personal Health Information Management System (PHIMS) supports activities such as acquisition, storage, organization, integration, and privacy-sensitive retrieval of consumer's health information. For usability and wide acceptance, the PHIMS should follow the design principles that guarantee privacy-aware health information sharing, individual information control, integration of information obtained from multiple medical IoT devices, health information security, and flexibility. Recently, blockchain technology has emerged as a lucrative option for the management of personal health information. In this paper, we propose eHealthChain-a blockchain-based PHIMS for managing health data originating from medical IoT devices and connected applications. The eHealthChain architecture consists of four layers, which are a blockchain layer for hosting a blockchain database, an IoT device layer for obtaining personal health data, an application layer for facilitating health data sharing, and an adapter layer, which interfaces the blockchain layer with an application layer. Compared to existing systems, eHealthChain provides complete control to the user in terms of personal health data acquisition, sharing, and self-management. We also present a detailed implementation of a Proof of Concept (PoC) prototype of eHealthChain system built using Hyperledger Fabric platform.
Enhanced Center Constraint Weighted A* Algorithm for Path Planning of Petrochemical Inspection Robot

In many practical applications of robot path planning, finding the shortest path is critical, while the response time is often overlooked but important. To address the problems of search node divergence and long calculation time in the A* routing algorithm in the large scenario, this paper presents a novel center constraint weighted A* algorithm (CCWA*). The heuristic function is modified to give different dynamic weights to nodes in different positions, and the node weights are changed in the specified direction during the expansion process, thereby reducing the number of search nodes. An adaptive threshold is further added to the heuristic function to enhance the adaptiveness of the algorithm. To verify the effectiveness of the CCWA* algorithm, simulations are performed on 2-dimensional grid maps of different sizes. The results show that the proposed algorithm speeds up the search process and reduces the planning time in the process of path planning in a multi-obstacle environment compared with the conventional A* algorithm and weighted A* algorithm.
Class-Aware Robust Adversarial Training for Object Detection

Object detection is an important computer vision task with plenty of real-world applications; therefore, how to enhance its robustness against adversarial attacks has emerged as a crucial issue. However, most of the previous defense methods focused on the classification task and had few analysis in the context of the object detection task. In this work, to address the issue, we present a novel class-aware robust adversarial training paradigm for the object detection task. For a given image, the proposed approach generates an universal adversarial perturbation to simultaneously attack all the occurred objects in the image through jointly maximizing the respective loss for each object. Meanwhile, instead of normalizing the total loss with the number of objects, the proposed approach decomposes the total loss into class-wise losses and normalizes each class loss using the number of objects for the class. The adversarial training based on the class weighted loss can not only balances the influence of each class but also effectively and evenly improves the adversarial robustness of trained models for all the object classes as compared with the previous defense methods. Furthermore, with the recent development of fast adversarial training, we provide a fast version of the proposed algorithm which can be trained faster than the traditional adversarial training while keeping comparable performance. With extensive experiments on the challenging PASCAL-VOC and MS-COCO datasets, the evaluation results demonstrate that the proposed defense methods can effectively enhance the robustness of the object detection models.
Recognition of pedestrian trajectories and attributes with computer vision and deep learning techniques

Analyzing the walking behavior of the public is vital for revealing the need for infrastructure design in a local neighborhood, supporting human-centric urban area development. Traditional walking behavior analysis practices relying on manual on-street surveys to collect pedestrian flow data are labor-intensive and tedious. On the contrary, automated video analytics using surveillance cameras based on computer vision and deep learning techniques appears more effective in generating pedestrian flow statistics. Nevertheless, most existing methods of pedestrian tracking and attribute recognition suffer from several challenging conditions, such as inter-person occlusion and appearance variations, which leads to ambiguous identities and hence inaccurate pedestrian flow statistics. Therefore, this paper proposes a more robust methodology of pedestrian tracking and attribute recognition, facilitating the analysis of pedestrian walking behavior. Specific limitations of a current state-of-the-art method are inferred, based on which several improvement strategies are proposed: 1) incorporating high-level pedestrian attributes to enhance pedestrian tracking, 2) a similarity measure integrating multiple cues for identity matching, and 3) a probation mechanism for more robust identity matching. From our evaluation using two public benchmark datasets, the developed strategies notably enhance the robustness of pedestrian tracking against the challenging conditions mentioned above. Subsequently, the outputs of trajectories and attributes are aggregated into fine-grained pedestrian flow statistics among different pedestrian groups. Overall, our developed framework can support a more comprehensive and reliable decision-making for human-centric planning and design in different urban areas. The framework is also applicable to exploiting pedestrian movement patterns in different scenes for analyses such as urban walkability evaluation. Moreover, the developed mechanisms are generalizable to future researches as a baseline, which provides generic insights of how to fundamentally enhance pedestrian tracking.
Critical parametric quantum sensing

Critical quantum systems are a promising resource for quantum metrology applications, due to the diverging susceptibility developed in proximity of phase transitions. Here, we assess the metrological power of parametric Kerr resonators undergoing driven-dissipative phase transitions. We fully characterize the quantum Fisher information for frequency estimation, and the Helstrom bound for frequency discrimination. By going beyond the asymptotic regime, we show that the Heisenberg precision can be achieved with experimentally reachable parameters. We design protocols that exploit the critical behavior of nonlinear resonators to enhance the precision of quantum magnetometers and the fidelity of superconducting qubit readout.
An Efficient Service Recommendation Algorithm for Cyber-Physical-Social Systems

Cyber-physical-social System (CPSS) technology closely integrates and coordinates computing resources, physical resources, and social information to provide humans with efficient and convenient services. Service recommendation technology satisfies the individual needs of users by collecting, processing, and analyzing user characteristic data. In recent years, service recommendation algorithms based on social networks and collaborative filtering have been widely used in CPSS. However, this type of service recommendation algorithm cannot effectively use the aliasing and auxiliary information in the service, nor can it generate different service recommendation schemes according to different scenarios. We propose a collaborative filtering service recommendation algorithm that combines heterogeneous information networks and topic models for CPSS. The proposed algorithm designs a more effective functional similarity measurement method through the word vectors learned by Word2vec. Then, we use the topic model to cluster the scenario-based service (SBS), thereby greatly reducing the time required for service recommendation. Experimental analysis results show that we finally determine the best combination of Mashup similarity between the two scenarios with better recommendation efficiency and accuracy. Thus, the proposed recommendation algorithm can effectively improve the quality of service of CPSS, and provide a basis for the personalized service of CPSS.
Isolated Sign Recognition from RGB Video using Pose Flow and Self-Attention

Automatic sign language recognition lies at the intersection of natural language processing (NLP) and computer vision. The highly successful transformer architectures, based on multi-head attention, originate from the field of NLP. The Video Transformer Network (VTN) is an adaptation of this concept for tasks that require video understanding, e.g., action recognition. However, due to the limited amount of labeled data that is commonly available for training automatic sign (language) recognition, the VTN cannot reach its full potential in this domain. In this work, we reduce the impact of this data limitation by automatically pre-extracting useful information from the sign language videos. In our approach, different types of information are offered to a VTN in a multi-modal setup. It includes per-frame human pose keypoints (extracted by OpenPose) to capture the body movement and hand crops to capture the (evolution of) hand shapes. We evaluate our method on the recently released AUTSL dataset for isolated sign recognition and obtain 92.92% accuracy on the test set using only RGB data. For comparison: the VTN architecture without hand crops and pose flow achieved 82% accuracy. A qualitative inspection of our model hints at further potential of multi-modal multi-head attention in a sign language recognition context.
MS-UDA: Multi-Spectral Unsupervised Domain Adaptation for Thermal Image Semantic Segmentation

In this letter, we propose a multi-spectral unsupervised domain adaptation for thermal image semantic segmentation. The proposed framework aims to address the data scarcity problem and boost segmentation performance in the thermal domain with the help of existing large-scale RGB datasets and segmentation knowledge from an RGB image segmentation network. We also enhance the generalization capability of our thermal segmentation network with pixel-level domain adaptation bridging day and night thermal image domains. With our framework, a thermal image segmentation network can achieve high performance without any ground-truth labels by exploiting successive multi-spectral knowledge transfers including RGB-to-RGB, RGB-to-Thermal, and Thermal-to-Thermal adaptations. Moreover, we provide a real-world RGB-Thermal semantic segmentation dataset with 950 manually annotated Cityscapes-style ground-truth labels in 19 classes. Experimental results on real-world datasets demonstrate the effectiveness and robustness of the proposed framework quantitatively and qualitatively.