Reviewerly/exHarmony huggingface dataset:
Speech gesture generation from the trimodal context of text, audio, and speaker identity

For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human-agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures. By incorporating a multimodal context and an adversarial training scheme, the proposed model outputs gestures that are human-like and that match with speech content and rhythm. We also introduce a new quantitative evaluation metric for gesture generation models. Experiments with the introduced metric and subjective human evaluation showed that the proposed gesture generation model is better than existing end-to-end generation models. We further confirm that our model is able to work with synthesized audio in a scenario where contexts are constrained, and show that different gesture styles can be generated for the same speech by specifying different speaker identities in the style embedding space that is learned from videos of various speakers. All the code and data is available at https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context.
Efficient and Privacy-Preserving Medical Research Support Platform Against COVID-19: A Blockchain-Based Approach

COVID-19 is a major global public health challenge and difficult to control in a short time completely. To prevent the COVID-19 epidemic from continuing to worsen, global scientific research institutions have actively carried out studies on COVID-19, thereby effectively improving the prevention, monitoring, tracking, control, and treatment of the epidemic. However, the COVID-19 electronic medical records (CEMRs) among hospitals worldwide are managed independently. With privacy consideration, CEMRs cannot be made public or shared, which is not conducive to in-depth and extensive research on COVID-19 by medical research institutions. In addition, even if new research results are developed, the disclosure and sharing process is slow. To address this issue, we propose a blockchain-based medical research support platform, which can provide efficient and privacy-preserving data sharing against COVID-19. First, hospitals and medical research institutions are treated as nodes on the alliance chain, so consensus and data sharing among the nodes is achieved. Then, COVID-19 patients, doctors, and researchers need to be authenticated in various institutes. Moreover, doctors and researchers need to be registered with the Fabric certificate authority. The CEMRs for COVID-19 patients uses the blockchain's pseudonym mechanism to protect privacy. After that, doctors upload CEMRs on the alliance chain, and researchers can obtain CEMRs from the alliance chain for research. Finally, the research results will be published on the blockchain for doctors to use. The experimental results show that the read and write performance and security performance on the alliance chain meet the requirements, which can promote the wide application of scientific research results against COVID-19.
Gesture interaction in virtual reality

With the development of virtual reality (VR) and human-computer interaction technology, how to use natural and efficient interaction methods in the virtual environment has become a hot topic of research. Gesture is one of the most important communication methods of human beings, which can effectively express users’ demands. In the past few decades, gesture-based interaction has made significant progress. This article focuses on the gesture interaction technology and discusses the definition and classification of gestures, input devices for gesture interaction, and gesture interaction recognition technology. The application of gesture interaction technology in virtual reality is studied, the existing problems in the current gesture interaction are summarized, and the future development is prospected.
Plan-and-Write: Towards Better Automatic Storytelling

Automatic storytelling is challenging since it requires generating long, coherent natural language to describes a sensible sequence of events. Despite considerable efforts on automatic story generation in the past, prior work either is restricted in plot planning, or can only generate stories in a narrow domain. In this paper, we explore open-domain story generation that writes stories given a title (topic) as input. We propose a plan-and-write hierarchical generation framework that first plans a storyline, and then generates a story based on the storyline. We compare two planning strategies. The dynamic schema interweaves story planning and its surface realization in text, while the static schema plans out the entire storyline before generating stories. Experiments show that with explicit storyline planning, the generated stories are more diverse, coherent, and on topic than those generated without creating a full plan, according to both automatic and human evaluations.
Qubitization of Arbitrary Basis Quantum Chemistry Leveraging Sparsity and Low Rank Factorization

Recent work has dramatically reduced the gate complexity required to quantum simulate chemistry by using linear combinations of unitaries based methods to exploit structure in the plane wave basis Coulomb operator. Here, we show that one can achieve similar scaling even for arbitrary basis sets (which can be hundreds of times more compact than plane waves) by using qubitized quantum walks in a fashion that takes advantage of structure in the Coulomb operator, either by directly exploiting sparseness, or via a low rank tensor factorization. We provide circuits for several variants of our algorithm (which all improve over the scaling of prior methods) including one with <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow class="MJX-TeXAtom-ORD"><mml:mover><mml:mi class="MJX-tex-caligraphic" mathvariant="script">O</mml:mi><mml:mo>~</mml:mo></mml:mover></mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:msup><mml:mi>N</mml:mi><mml:mrow class="MJX-TeXAtom-ORD"><mml:mn>3</mml:mn><mml:mrow class="MJX-TeXAtom-ORD"><mml:mo>/</mml:mo></mml:mrow><mml:mn>2</mml:mn></mml:mrow></mml:msup><mml:mi>λ</mml:mi><mml:mo stretchy="false">)</mml:mo></mml:math> T complexity, where <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>N</mml:mi></mml:math> is number of orbitals and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>λ</mml:mi></mml:math> is the 1-norm of the chemistry Hamiltonian. We deploy our algorithms to simulate the FeMoco molecule (relevant to Nitrogen fixation) and obtain circuits requiring about seven hundred times less surface code spacetime volume than prior quantum algorithms for this system, despite us using a larger and more accurate active space.
An exploratory study of smart contracts in the Ethereum blockchain platform

Ethereum is a blockchain platform that supports smart contracts. Smart contracts are pieces of code that perform general-purpose computations. For instance, smart contracts have been used to implement crowdfunding initiatives that raised a total of US$6.2 billion from January to June of 2018. In this paper, we conduct an exploratory study of smart contracts. Differently from prior studies that focused on particular aspects of a subset of smart contracts, our goal is to have a broader understanding of all contracts that are currently deployed in Ethereum. In particular, we elucidate how frequently used the contracts are (activity level), what they do (category), and how complex they are (source code complexity). To conduct this study, we mined and cross-linked data from four sources: Ethereum dataset on the Google BigQuery platform, Etherscan, State of the DApps, and CoinMarketCap. Our study period runs from July 2015 (inception of Ethereum) until September 2018. With regards to activity level, we notice that it is concentrated on a very small subset of the contracts. More specifically, only 0.05% of the smart contracts are the target of 80% of the transactions that are sent to contracts. New solutions to cope with Ethereum’s limited scalability should take such an activity imbalance into consideration. With regards to categories, we highlight that the new and widely advertised rich programming model of smart contracts is currently being used to develop very simple applications that tend to be token-centric (e.g., ICOs, Crowdsales, etc). Finally, with regards to code complexity, we observe that the source code of high-activity verified contracts is small, with at most 211 instructions in 80% of the cases. These contracts also commonly include at least two subcontracts and libraries in their source code. The comment ratio of these contracts is also significantly higher than that of GitHub top-starred projects written in Java, C++, and C#. Hence, the source code of high-activity verified smart contracts exhibit particular complexity characteristics compared to other popular programming languages. Further studies are necessary to uncover the actual reasons behind such differences. Finally, based on our findings, we propose an open research agenda to drive and foster future studies in the area.
Analyzing User-Level Privacy Attack Against Federated Learning

Federated learning has emerged as an advanced privacy-preserving learning technique for mobile edge computing, where the model is trained in a decentralized manner by the clients, preventing the server from directly accessing those private data from the clients. This learning mechanism significantly challenges the attack from the server side. Although the state-of-the-art attacking techniques that incorporated the advance of Generative adversarial networks (GANs) could construct class representatives of the global data distribution among all clients, it is still challenging to distinguishably attack a specific client (i.e., user-level privacy leakage), which is a stronger privacy threat to precisely recover the private data from a specific client. To analyze the privacy leakage of federated learning, this paper gives the first attempt to explore user-level privacy leakage by the attack from a malicious server. We propose a framework incorporating GAN with a multi-task discriminator, called multi-task GAN - Auxiliary Identification (mGAN-AI), which simultaneously discriminates category, reality, and client identity of input samples. The novel discrimination on client identity enables the generator to recover user specified private data. Unlike existing works interfering the federated learning process, the proposed method works “invisibly” on the server side. Furthermore, considering the anonymization strategy for mitigating mGAN-AI, we propose a beforehand linkability attack which re-identifies the anonymized updates by associating the client representatives. A novel siamese network fusing the identification and verification models is developed for measuring the similarity of representatives. The experimental results demonstrate the effectiveness of the proposed approaches and the superior to the state-of-the-art.
Information-Theoretic Probing for Linguistic Structure

The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually "know" about natural language.Probes are a natural way of assessing this.When probing, a researcher chooses a linguistic task and trains a supervised model to predict annotations in that linguistic task from the network's learned representations.If the probe does well, the researcher may conclude that the representations encode knowledge related to the task.A commonly held belief is that using simpler models as probes is better; the logic is that simpler models will identify linguistic structure, but not learn the task itself.We propose an information-theoretic operationalization of probing as estimating mutual information that contradicts this received wisdom: one should always select the highest performing probe one can, even if it is more complex, since it will result in a tighter estimate, and thus reveal more of the linguistic information inherent in the representation.The experimental portion of our paper focuses on empirically estimating the mutual information between a linguistic property and BERT, comparing these estimates to several baselines.We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research-plus Englishtotalling eleven languages.Our implementation is available in https://github.com/rycolab/info-theoretic-probing.
Vector Quantized Diffusion Model for Text-to-Image Synthesis

We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-to-image generation tasks because it not only eliminates the unidirectional bias with existing methods but also allows us to incorporate a mask-and-replace diffusion strategy to avoid the accumulation of errors, which is a serious problem with existing methods. Our experiments show that the VQ-Diffusion produces significantly better text-to-image generation results when compared with conventional autoregressive (AR) models with similar numbers of parameters. Compared with previous GAN-based text-to-image methods, our VQ-Diffusion can handle more complex scenes and improve the synthesized image quality by a large margin. Finally, we show that the image generation computation in our method can be made highly efficient by reparameterization. With traditional AR methods, the text-to-image generation time increases linearly with the output image resolution and hence is quite time consuming even for normal size images. The VQ-Diffusion allows us to achieve a better trade-off between quality and speed. Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality. The code and models are available at https://github.com/cientgu/VQ-Diffusion.
International Sentiment Analysis for News and Blogs

There is a growing interest in mining opinions using sentiment analysis methods from sources such as news, blogs and product reviews. Most of these methods have been developed for English and are difficult to generalize to other languages. We explore an approach utilizing state-of-the-art machine translation technology and perform sentiment analysis on the English translation of a foreign language text. Our experiments indicate that (a) entity sentiment scores obtained by our method are statistically significantly correlated across nine languages of news sources and five languages of a parallel corpus; (b) the quality of our sentiment analysis method is largely translator independent; (c) after applying certain normalization techniques, our entity sentiment scores can be used to perform meaningful cross-cultural comparisons.
Secure Key Agreement and Key Protection for Mobile Device User Authentication

As mobile devices ownership becomes more prevalent (e.g., a user owns multiple mobile devices), the capability to offer secure and user friendly authentication becomes increasingly important. A large number of identity-based user authentication mechanisms for the wireless mobile environment have been proposed. However, they are not generally designed for situations where a user's private key and some other sensitive data can be exposed if his/her mobile device is remotely or physically controlled by an attacker. Threshold secret sharing is one of the solutions to this problem, but it is limited in the requirement that there should exist an honest third-party to hold the complete key after the secret reconstruction process. Therefore, in this paper, we consider the special case that only two devices (i.e., no honest party) at the user's side jointly perform user authentication with a server, and neither device can successfully complete the authentication process alone. Moreover, the key reconstruction is not needed during authentication so that neither device can hold a complete key. We then analyze the security of the proposed protocol and show that it satisfies all known security requirements in practical applications, particularly the key exposure attack resistance. The performance analysis of the proposed protocol is also presented to demonstrate its practicality.
General Resource Theories in Quantum Mechanics and Beyond: Operational Characterization via Discrimination Tasks

A unifying framework for quantum resources shows that the utility of many physical phenomena in quantum information processing tasks can be described in a common formalism based on state and channel discrimination.
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

Machine comprehension of texts longer than a single sentence often requires coreference resolution.However, most current reading comprehension benchmarks do not contain complex coreferential phenomena and hence fail to evaluate the ability of models to resolve coreference.We present a new crowdsourced dataset containing more than 24K span-selection questions that require resolving coreference among entities in over 4.7K English paragraphs from Wikipedia.Obtaining questions focused on such phenomena is challenging, because it is hard to avoid lexical cues that shortcut complex reasoning.We deal with this issue by using a strong baseline model as an adversary in the crowdsourcing loop, which helps crowdworkers avoid writing questions with exploitable surface cues.We show that state-of-the-art reading comprehension models perform significantly worse than humans on this benchmark-the best model performance is 70.5 F 1 , while the estimated human performance is 93.4 F 1 .
Adversarial-Learned Loss for Domain Adaptation

Recently, remarkable progress has been made in learning transferable representation across domains. Previous works in domain adaptation are majorly based on two techniques: domain-adversarial learning and self-training. However, domain-adversarial learning only aligns feature distributions between domains but does not consider whether the target features are discriminative. On the other hand, self-training utilizes the model predictions to enhance the discrimination of target features, but it is unable to explicitly align domain distributions. In order to combine the strengths of these two methods, we propose a novel method called Adversarial-Learned Loss for Domain Adaptation (ALDA). We first analyze the pseudo-label method, a typical self-training method. Nevertheless, there is a gap between pseudo-labels and the ground truth, which can cause incorrect training. Thus we introduce the confusion matrix, which is learned through an adversarial manner in ALDA, to reduce the gap and align the feature distributions. Finally, a new loss function is auto-constructed from the learned confusion matrix, which serves as the loss for unlabeled target samples. Our ALDA outperforms state-of-the-art approaches in four standard domain adaptation datasets. Our code is available at https://github.com/ZJULearning/ALDA.
Forecasting artificial intelligence on online customer assistance: Evidence from chatbot patents analysis

The main goal of this research is to provide a comprehensive understanding of the actual progresses in artificial intelligence, with emphasis on chatbots as emerging forms of customer assistance in online retailing. Drawing upon an analysis of the chatbot patents in the past 20 years, our findings show the increasing technology push towards the adoption of new conversational agents based on natural language. Findings also highlight the extent to which the research and development efforts are attempting to improve artificial intelligence systems that characterize chatbots. To this end, technology advancements are mainly focusing on: (i) improving chatbot ability to automatically draw inferences on users starting from multiple data sources, and (ii) using consumers’ knowledge adaptively to provide more customized solutions. Finally, results show the tight relationship between the digital assistants’ analytical skills and their ability to automatically interact with the users.
An intelligent cooperative mission planning scheme of UAV swarm in uncertain dynamic environment

This paper presents an intelligent cooperative mission planning scheme for unmanned aerial vehicle (UAV) swarm, to search and attack the time-sensitive moving targets in uncertain dynamic environment, by using a hybrid artificial potential field and ant colony optimization (HAPF-ACO) method. In the search-attack mission environment of UAV swarm under the dynamic topology interaction, a time-sensitive target probability map is established. Based on the HAPF, the target attraction field, threat repulsive field and repulsive field are constructed for the environmental cognition. A distributed ACO algorithm is designed to improve the UAVs' global searching capability. For this mission planning problem, four time-sensitive moving target types and four constraint types of UAV swarm are considered, which will contribute to the practical applications of the HAPF-ACO. Several simulations are carried out to exhibit the superiority on the task execution efficiency and obstacle and collision avoidance performance of the proposed intelligent cooperative mission planning scheme.
Design of optical neural networks with component imprecisions

For the benefit of designing scalable, fault resistant optical neural networks (ONNs), we investigate the effects architectural designs have on the ONNs' robustness to imprecise components.We train two ONNs -one with a more tunable design (GridNet) and one with better fault tolerance (FFTNet) -to classify handwritten digits.When simulated without any imperfections, GridNet yields a better accuracy (∼ 98%) than FFTNet (∼ 95%).However, under a small amount of error in their photonic components, the more fault tolerant FFTNet overtakes GridNet.We further provide thorough quantitative and qualitative analyses of ONNs' sensitivity to varying levels and types of imprecisions.Our results offer guidelines for the principled design of fault-tolerant ONNs as well as a foundation for further research.
Overview and Comparison of Gate Level Quantum Software Platforms

Quantum computers are available to use over the cloud, but the recent explosion of quantum software platforms can be overwhelming for those deciding on which to use. In this paper, we provide a current picture of the rapidly evolving quantum computing landscape by comparing four software platforms - Forest (pyQuil), Qiskit, ProjectQ, and the Quantum Developer Kit (Q#) - that enable researchers to use real and simulated quantum devices. Our analysis covers requirements and installation, language syntax through example programs, library support, and quantum simulator capabilities for each platform. For platforms that have quantum computer support, we compare hardware, quantum assembly languages, and quantum compilers. We conclude by covering features of each and briefly mentioning other quantum computing software packages.