Reviewerly/exHarmony huggingface dataset:
Walking with MIND

The EmbodiedQA is a task of training an embodied agent by intelligently navigating in a simulated environment and gathering visual information to answer questions. Existing approaches fail to explicitly model the mental imagery function of the agent, while the mental imagery is crucial to embodied cognition, and has a close relation to many high-level meta-skills such as generalization and interpretation. In this paper, we propose a novel Mental Imagery eNhanceD (MIND) module for the embodied agent, as well as a relevant deep reinforcement framework for training. The MIND module can not only model the dynamics of the environment (e.g. 'what might happen if the agent passes through a door') but also help the agent to create a better understanding of the environment (e.g. 'The refrigerator is usually in the kitchen'). Such knowledge makes the agent a faster and better learner in locating a feasible policy with only a few trails. Furthermore, the MIND module can generate mental images that are treated as short-term subgoals by our proposed deep reinforcement framework. These mental images facilitate policy learning since short-term subgoals are easy to achieve and reusable. This yields better planning efficiency than other algorithms that learn a policy directly from primitive actions. Finally, the mental images visualize the agent's intentions in a way that human can understand, and this endows our agent's actions with more interpretability. The experimental results and further analysis prove that the agent with the MIND module is superior to its counterparts not only in EQA performance but in many other aspects such as route planning, behavioral interpretation, and the ability to generalize from a few examples.
Still Not Readable? An Interactive Tool for Recommending Color Pairs with Sufficient Contrast based on Existing Visual Designs

Most contrast checking tools allow designers to verify if a set of colors satisfy the minimum contrast requirements for text and background colors. However, such tools usually do not provide the designer with any clues on how the colors should be adjusted to ensure enough contrast. This demo shows our web-based contrast tool which suggests how existing set of colors could be adjusted to satisfy minimum contrast requirements while maintaining the gist of the original visual design.
Effective Sentiment-relevant Word Selection for Multi-modal Sentiment Analysis in Spoken Language

Computational modeling of human spoken language is an emerging research area in multimedia analysis spanning across the text and acoustic modalities. Multi-modal sentiment analysis is one of the most fundamental tasks in human spoken language understanding. In this paper, we propose a novel approach to selecting effective sentiment-relevant words for multi-modal sentiment analysis with focus on both the textual and acoustic modalities. Unlike the conventional soft attention mechanism, we employ a deep reinforcement learning mechanism to perform sentiment-relevant word selection and fully remove invalid words of each modality for multi-modal sentiment analysis. Specifically, we first align the raw text and audio at the word level and extract independent handcraft features for each modality to yield the textual and acoustic word sequence. Second, we establish two collaborative agents to deal with the textual and acoustic modalities in spoken language respectively. On this basis, we formulate the sentiment-relevant word selection process in a multi-modal setting as a multi-agent sequential decision problem and solve it with a multi-agent reinforcement learning approach. Detailed evaluations of multi-modal sentiment classification and emotion recognition on three benchmark datasets demonstrate the great effectiveness of our approach over several conventional competitive baselines.
Quasi-Globally Optimal and Efficient Vanishing Point Estimation in Manhattan World

The image lines projected from parallel 3D lines intersect at a common point called the vanishing point (VP). Manhattan world holds for the scenes with three orthogonal VPs. In Manhattan world, given several lines in a calibrated image, we aim at clustering them by three unknown-but-sought VPs. The VP estimation can be reformulated as computing the rotation between the Manhattan frame and the camera frame. To compute this rotation, state-of-the-art methods are based on either data sampling or parameter search, and they fail to guarantee the accuracy and efficiency simultaneously. In contrast, we propose to hybridize these two strategies. We first compute two degrees of freedom (DOF) of the above rotation by two sampled image lines, and then search for the optimal third DOF based on the branch-and-bound. Our sampling accelerates our search by reducing the search space and simplifying the bound computation. Our search is not sensitive to noise and achieves quasi-global optimality in terms of maximizing the number of inliers. Experiments on synthetic and real-world images showed that our method outperforms state-of-the-art approaches in terms of accuracy and/or efficiency.
A nonlocal game for witnessing quantum networks

Abstract Nonlocal game as a witness of the nonlocality of entanglement is of fundamental importance in various fields. The well-known nonlocal games or equivalent linear Bell inequalities are only useful for Bell networks consisting of single entanglement. Our goal in this paper is to propose a unified method for constructing cooperating games in network scenarios. We propose an efficient method to construct multipartite nonlocal games from any graphs. The main idea is the graph representation of entanglement-based quantum networks. We further specify these graphic games with quantum advantages by providing a simple sufficient and necessary condition. The graphic games imply a linear Bell testing of the nonlocality of general quantum networks consisting of EPR states. It also allows generating new instances going beyond CHSH game. These results have interesting applications in quantum networks, Bell theory, computational complexity, and theoretical computer science.
Prediction-CGAN

The underlying challenge of human action prediction, i.e. maintaining prediction accuracy at very beginning of an action execution, is still not well handled. In this paper, we propose a Prediction Conditional Generative Adversarial Network (Prediction-CGAN) for predicting action, which shares information between completely observed and partially observed videos. Instead of generating future frames, we aim at completing visual representations of unfinished video, which can be directly utilized to predict action label no matter at any progress levels. The Prediction-CGAN incorporates the completion constraint to learn a transformation from incomplete actions to complete actions; the adversarial constraint to ensure the generation has similar discriminative power to complete representation; the label consistency constraint to encourage label consistency between each segment and its corresponding complete video; and the confidence monotonically increasing constraint to yield increasingly accurate predictions as observing more frames. Meanwhile, we introduce a novel adversarial criterion especially for prediction task, which requires the generation is more discriminative than its corresponding incomplete representation, while the generation is less discriminative than its real complete representation. In experiments, we present adequate evaluations to show that the proposed Prediction-CGAN outperforms state-of-the-art methods in action prediction.
Effective and fast binarization method for combined degradation on ancient documents

Document image binarization is a challenging task because of combined degradation in a document. In this study, a new binarization method is proposed for binarizing an ancient document with combined degradation. The proposed method comprises the following four stages: histogram analysis, contrast enhancement, local adaptive thresholding, and artifact removal. In histogram analysis, a new approach is applied to establish a uniform background. Next, the image contrast is enhanced using a new contrast enhancement, and then the document is binarized using a novel local adaptive thresholding. Artifacts from the binarization process are removed in the artifact removal stage. Finally, an experiment is conducted using one private and four public datasets and by simulating the proposed method with and without contrast enhancement. The results showed that the proposed method is faster and more effective compared to other state-of-the-art procedures for binarizing ancient documents.
A cooperative target search method based on intelligent water drops algorithm

This study investigates the problem of cooperative search path planning for multiple unmanned aerial vehicles. Firstly, we present a search probability map based environmental model, and formulate a multi-UAV cooperative search path optimization problem. Then, we propose a solution strategy based on an improved intelligent water drops (IWD) optimization algorithm. Compared with the traditional IWD algorithm, the proposed algorithm can simultaneously model the cooperation and competition among UAVs by introducing several heterogeneous water drop populations and a co-evolutionary mechanism. Water drops in the same population cooperate with each other, whereas water drops among different populations compete with each other to obtain the optimal path and reduce meaningless search effort. Meanwhile, the proposed algorithm can improve its search capability using a new soil update mechanism. Simulation results demonstrate the advantages of the proposed method over other baselines.
Studying Transaction Fees in the Bitcoin Blockchain with Probabilistic Logic Programming

In Bitcoin, if a miner is able to solve a computationally hard problem called proof of work, it will receive an amount of bitcoin as a reward which is the sum of the fees for the transactions included in a block plus an amount inversely proportional to the number of blocks discovered so far. At the moment of writing, the block reward is several orders of magnitude greater than the sum of transaction fees. Usually, miners try to collect the largest reward by including transactions associated with high fees. The main purpose of transaction fees is to prevent network spamming. However, they are also used to prioritize transactions. In order to use the minimum amount of fees, users usually have to find a compromise between fees and urgency of a transaction. In this paper, we develop a probabilistic logic model to experimentally analyze how fees affect confirmation time and miner’s revenue and to predict if an increase of average fees will generate a situation when the miner gets more reward by not following the protocol.
Particle method for segmentation of breast tumors in ultrasound images

We propose a new segmentation method based on multiple walking particles (WP) bouncing from the image edges. The particles are able to segment objects characterized by deep concavities as narrow as one pixel and handle single or multiple objects characterized by a noisy background and broken boundaries (“weak edge”, “boundary leakage”). The particles are designed to segment the image by permanently staying inside the object and repairing the boundaries where necessary. The proposed WP combine the advantages of the continuous diffusion models with the principles of multi-agent systems. WP have been tested against recent active contours and the distance regularized level set method on a set of complex-shaped synthetic images and ultrasound (US) images of breast cancer (http://onlinemedicalimages.com). The method has also been compared with localizing region-based active contours, the fuzzy C-mean level set method, and morphological active contours. The WP are faster and more accurate for images characterized by low contrast, noise, broken boundaries, or boundary leakage. However, for good quality, simple shaped objects the WP work similarly to the conventional methods. There is still an important difference even in this case: the WP do not require initialization. A video demo of the algorithm is at https://drive.google.com/drive/folders/1SlTphINKtdUwvdjjxiakFrwI2dDlIUAU
AIRD: Adversarial Learning Framework for Image Repurposing Detection

Image repurposing is a commonly used method for spreading misinformation on social media and online forums, which involves publishing untampered images with modified metadata to create rumors and further propaganda. While manual verification is possible, given vast amounts of verified knowledge available on the internet, the increasing prevalence and ease of this form of semantic manipulation call for the development of robust automatic ways of assessing the semantic integrity of multimedia data. In this paper, we present a novel method for image repurposing detection that is based on the real-world adversarial interplay between a bad actor who repurposes images with counterfeit metadata and a watchdog who verifies the semantic consistency between images and their accompanying metadata, where both players have access to a reference dataset of verified content, which they can use to achieve their goals. The proposed method exhibits state-of-the-art performance on location-identity, subject-identity and painting-artist verification, showing its efficacy across a diverse set of scenarios.
Tackling Data Inefficiency: Compressing the Bitcoin Blockchain

In blockchain-based solutions, the amount of data stored in the blockchain increases steadily. Considerable amounts of data accordingly need to be download and stored at decentralized nodes to carry out meaningful validations. For the Bitcoin blockchain, approximately 197 GB must currently be downloaded and processed by any node aiming to fully verify the correctness of stored transactions. This represents an emerging problem with respect to resource-constrained thin clients that lack the required download or storage capacities, but with which meaningful validations still need to be carried out. This problem will gain even more relevance in the future, as blockchain technology is applied to new domains such as the Internet of Things (IoT) that necessitate the use of thin clients. We address this emerging problem by proposing a novel compressible blockchain architecture. In our proposal, we use a snapshot-based approach, to create snapshots of the blockchain at regular intervals and store them in blocks. These blocks are chained, forming a second blockchain that is linked to the primary chain. In this way, the amount of data that needs to be downloaded and stored by decentralized nodes is reduced considerably, while the full validation of the entire blockchain still remains feasible. In this paper, we describe the proposed compressible blockchain architecture in detail. We conducted evaluations that demonstrate the feasibility of the proposed solution and show its advantages over related approaches introduced in the literature. Overall, our design can be used to reduce the size of the blockchain by up to 93%, facilitating secure blockchain-based applications even on resource-constrained thin clients such as IoT devices.
Networked Emotional News on Social Media

This paper examines news and emotions by studying Facebook Reactions (emojis) that were introduced in February 2016. Four main emotions are investigated by exploring readers' reactions to news stories selected from a sample of over 12,000 news items posted on the English language Facebook pages of CNN, ABC News, BBC, and Sky News in 2016. There is little empirical research on the emotional aspect of news consumption on social media, and this study proposes the concept of networked emotional news and links it to research on emotions and the theory of news values. The findings indicate that social media readers are emotionally engaged with news that involves positive feelings especially love if one takes into account the top news stories as well as the total number of emoji clicks for the top 10 stories, while negative news that involves sad feelings remain the most dominant if the overall frequency of such stories is taken into account.
A Unifying Model of Legal Smart Contracts

Legal smart contracts have been a subject of research for decades, especially since the fusion of deontic logic with traditional programming poses significant challenges. The issue of how to develop and verify legal smart contracts is growing in importance, not least due to the rapid adoption of blockchain-based smart contracts. In this paper, we want to pave the way towards a model-driven approach at legal smart contract development. To this end, we combine insights from literature in law and legal informatics with capabilities of existing modeling approaches and give a unifying model that encapsulates essential components of legal smart contracts. The unifying model may be used as a reference for language designers aiming at a holistic representation of legal smart contracts in a model-driven architecture. It may further serve as a basis for comparing existing modeling frameworks, which we demonstrate by applying it to a set of eight distinct languages.
Gesture recognition for human–machine interaction in table tennis video based on deep semantic understanding

The analysis of moving objects in videos, especially the recognition of human motions and gestures, is attracting increasing emphasis in computer vision area. However, most existing video analysis methods do not take into account the effect of video semantic information. The topological information of the video image plays an important role in describing the association relationship of the image content, which will help to improve the discriminability of the video feature expression. Based on the above considerations, we propose a video semantic feature learning method that integrates image topological sparse coding with dynamic time warping algorithm to improve the gesture recognition in videos. This method divides video feature learning into two phases: semi-supervised video image feature learning and supervised optimization of video sequence features. Next, a distance weighting based dynamic time warping algorithm and K-nearest neighbor algorithm is leveraged to recognize gestures. We conduct comparative experiments on table tennis video dataset. The experimental results show that the proposed method is more discriminative to the expression of video features and can effectively improve the recognition rate of gestures in sports video.
Multi-programming language software systems modularization

A software system evolves over time to meet the user's new requirements as well as to adapt to the environment. This causes it deviates from its original and documented structure. Hence, after a while, due to its low understandability, making new changes will not be easy. Modularization is utilized to extract the software system structure from the source code to understand it. Recently, there has been a significant increase in writing large-scale applications in collaboration with several programming languages; for example, Mozilla Firefox 3.7 is written in more than twenty programming languages. Understanding such applications for maintaining them is a challenging problem. Existing modularization methods to understand an application are only applicable to programs which are written in a programming language. This paper introduces the concept of multi-programming language modularization. It then presents a technique for modularization of applications which are written in multiple programming languages. The results on Mozilla Firefox demonstrate that the proposed approach can extract a modularization close to human experts, a capability which may help in understanding an application.
Privacy in Mobile Sensing

In this chapter, we discuss the privacy implications of mobile sensing and modern psycho-social sciences. We aim to raise awareness of the multifaceted nature of privacy, describing the legal, technical and applied aspects in some detail. Not only since the European GDPR, these aspects lead to a broad spectrum of challenges of which data processors cannot be absolved by a simple consent form from their users. Instead appropriate technical and organizational measures should be put in place through a proper privacy engineering process. Throughout the chapter, we illustrate the importance of privacy protection through a set of examples and also technical approaches to address these challenges. We conclude this chapter with an outlook on privacy in mobile sensing, digital phenotyping and, psychoinformatics.
Progressive Operational Perceptrons with Memory

Generalized Operational Perceptron (GOP) was proposed to generalize the linear neuron model used in the traditional Multilayer Perceptron (MLP) by mimicking the synaptic connections of biological neurons showing nonlinear neurochemical behaviours. Previously, Progressive Operational Perceptron (POP) was proposed to train a multilayer network of GOPs which is formed layer-wise in a progressive manner. While achieving superior learning performance over other types of networks, POP has a high computational complexity. In this work, we propose POPfast, an improved variant of POP that signicantly reduces the computational complexity of POP, thus accelerating the training time of GOP networks. In addition, we also propose major architectural modications of POPfast that can augment the progressive learning process of POP by incorporating an information preserving, linear projection path from the input to the output layer at each progressive step. The proposed extensions can be interpreted as a mechanism that provides direct information extracted from the previously learned layers to the network, hence the term “memory”. This allows the network to learn deeper architectures and better data representations. An extensive set of experiments in human action, object, facial identity and scene recognition problems demonstrates that the proposed algorithms can train GOP networks much faster than POPs while achieving better performance compared to original POPs and other related algorithms.
Super-resolution of Omnidirectional Images Using Adversarial Learning

An omnidirectional image (ODI) enables viewers to look in every direction from a fixed point through a head-mounted display providing an immersive experience compared to that of a standard image. Designing immersive virtual reality systems with ODIs is challenging as they require high resolution content. In this paper, we study super-resolution for ODIs and propose an improved generative adversarial network based model which is optimized to handle the artifacts obtained in the spherical observational space. Specifically, we propose to use a fast PatchGAN discriminator, as it needs fewer parameters and improves the super-resolution at a fine scale. We also explore the generative models with adversarial learning by introducing a spherical-content specific loss function, called 360-SS. To train and test the performance of our proposed model we prepare a dataset of 4500 ODIs. Our results demonstrate the efficacy of the proposed method and identify new challenges in ODI super-resolution for future investigations.