Reviewerly/exHarmony huggingface dataset:
Multimodal MRI synthesis using unified generative adversarial networks

Purpose Complementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time‐consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis. Methods A unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1‐weighted (T1), T1‐weighted and contrast‐enhanced (T1c), T2‐weighted (T2), and fluid‐attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal‐to‐noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE). Results The proposed model was trained and tested on a cohort of 274 glioma patients with well‐aligned multi‐types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034 ± 0.005, 0.041 ± 0.006, and 0.041 ± 0.006, the PSNRs respectively are 32.353 ± 2.525 dB, 30.016 ± 2.577 dB, and 29.091 ± 2.795 dB, the SSIMs are 0.974 ± 0.059, 0.969 ± 0.059, and 0.959 ± 0.059, the VIF are 0.750 ± 0.087, 0.706 ± 0.097, and 0.654 ± 0.062, and NIQE are 1.396 ± 0.401, 1.511 ± 0.460, and 1.259 ± 0.358, respectively. Conclusions We proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.
Regularized Two-Branch Proposal Networks for Weakly-Supervised Moment Retrieval in Videos

Video moment retrieval aims to localize the target moment in an video according to the given sentence. The weak-supervised setting only provides the video-level sentence annotations during training. Most existing weak-supervised methods apply a MIL-based framework to develop inter-sample confrontment, but ignore the intra-sample confrontment between moments with semantically similar contents. Thus, these methods fail to distinguish the target moment from plausible negative moments. In this paper, we propose a novel Regularized Two-Branch Proposal Network to simultaneously consider the inter-sample and intra-sample confrontments. Concretely, we first devise a language-aware filter to generate an enhanced video stream and a suppressed video stream. We then design the sharable two-branch proposal module to generate positive proposals from the enhanced stream and plausible negative proposals from the suppressed one for sufficient confrontment. Further, we apply the proposal regularization to stabilize the training process and improve model performance. The extensive experiments show the effectiveness of our method. Our code is released at here.
Deep Graph Matching and Searching for Semantic Code Retrieval

Code retrieval is to find the code snippet from a large corpus of source code repositories that highly matches the query of natural language description. Recent work mainly uses natural language processing techniques to process both query texts (i.e., human natural language) and code snippets (i.e., machine programming language), however neglecting the deep structured features of query texts and source codes, both of which contain rich semantic information. In this paper, we propose an end-to-end deep graph matching and searching (DGMS) model based on graph neural networks for the task of semantic code retrieval. To this end, we first represent both natural language query texts and programming language code snippets with the unified graph-structured data, and then use the proposed graph matching and searching model to retrieve the best matching code snippet. In particular, DGMS not only captures more structural information for individual query texts or code snippets but also learns the fine-grained similarity between them by cross-attention based semantic matching operations. We evaluate the proposed DGMS model on two public code retrieval datasets with two representative programming languages (i.e., Java and Python). Experiment results demonstrate that DGMS significantly outperforms state-of-the-art baseline models by a large margin on both datasets. Moreover, our extensive ablation studies systematically investigate and illustrate the impact of each part of DGMS.
Bootstrapping Conditional GANs for Video Game Level Generation

Generative Adversarial Networks (GANs) have shown impressive results for image generation. However, GANs face challenges in generating contents with certain types of constraints, such as game levels. Specifically, it is difficult to generate levels that have aesthetic appeal and are playable at the same time. Additionally, because training data usually is limited, it is challenging to generate unique levels with current GANs. In this paper, we propose a new GAN architecture named Conditional Embedding Self-Attention Generative Adversarial Net-work (CESAGAN) and a new bootstrapping training procedure. The CESAGAN is a modification of the self-attention GAN that incorporates an embedding feature vector input to condition the training of the discriminator and generator. This allows the network to model non-local dependency between game objects, and to count objects. Additionally, to reduce the number of levels necessary to train the GAN, we propose a bootstrapping mechanism in which playable generated levels are added to the training set. The results demonstrate that the new approach does not only generate a larger number of levels that are playable but also generates fewer duplicate levels compared to a standard GAN.
ADGAN: Protect Your Location Privacy in Camera Data of Auto-Driving Vehicles

Computer vision and deep neural networks have been significantly promoting the development of visual perception in these years. Particularly, for autonomous vehicles, real-time image/video data is captured by onboard cameras and analyzed by computer vision techniques in many real applications. In the captured camera data, some contents can be used as auxiliary information to infer individuals' locations and trajectories, which leads to severe privacy leakage but has been rarely studied. Thus, the goal of this article is to protect individuals' location privacy by hiding side-channel information in the captured data while preserving the data utility for downstream applications. To this end, the technology of generative adversarial networks (GAN) is utilized to design two novel models, named ADGAN-I and ADGAN-II, both of which can take the original camera data as inputs and generate privacy-preserving outputs according to predefined sensitive object class. Thus, the processed camera data can defend location inference attack from adversaries in offline applications. Moreover, in ADGAN-I and ADGAN-II, the tradeoff between location privacy and data utility can be effectively balanced. Finally, the results of extensive real-data experiments validate the superiority of our proposed models over the state of the arts in utility preservation and privacy protection for autonomous vehicles' images and videos.
Hate begets Hate

With the ongoing debate on 'freedom of speech' vs. 'hate speech,' there is an urgent need to carefully understand the consequences of the inevitable culmination of the two, i.e., 'freedom of hate speech' over time. An ideal scenario to understand this would be to observe the effects of hate speech in an (almost) unrestricted environment. Hence, we perform the first temporal analysis of hate speech on Gab.com, a social media site with very loose moderation policy. We first generate temporal snapshots of Gab from millions of posts and users. Using these temporal snapshots, we compute an activity vector based on DeGroot model to identify hateful users. The amount of hate speech in Gab is steadily increasing and the new users are becoming hateful at an increased and faster rate. Further, our analysis analysis reveals that the hate users are occupying the prominent positions in the Gab network. Also, the language used by the community as a whole seem to correlate more with that of the hateful users as compared to the non-hateful ones. We discuss how, many crucial design questions in CSCW open up from our work.
B2

Data scientists have embraced computational notebooks to author analysis code and accompanying visualizations within a single document. Currently, although these media may be interleaved, they remain siloed: interactive visualizations must be manually specified as they are divorced from the analysis provenance expressed via dataframes, while code cells have no access to users' interactions with visualizations, and hence no way to operate on the results of interaction. To bridge this divide, we present B2, a set of techniques grounded in treating data queries as a shared representation between the code and interactive visualizations. B2 instruments data frames to track the queries expressed in code and synthesize corresponding visualizations. These visualizations are displayed in a dashboard to facilitate interactive analysis. When an interaction occurs, B2 reifies it as a data query and generates a history log in a new code cell. Subsequent cells can use this log to further analyze interaction results and, when marked as reactive, to ensure that code is automatically recomputed when new interaction occurs. In an evaluative study with data scientists, we find that B2 promotes a tighter feedback loop between coding and interacting with visualizations. All participants frequently moved from code to visualization and vice-versa, which facilitated their exploratory data analysis in the notebook.
Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots

This book provides a comprehensive introduction to Conversational AI. While the idea of interacting with a computer using voice or text goes back a long way, it is only in recent years that this idea has become a reality with the emergence of digital personal assistants, smart speakers, and chatbots. Advances in AI, particularly in deep learning, along with the availability of massive computing power and vast amounts of data, have led to a new generation of dialogue systems and conversational interfaces. Current research in Conversational AI focuses mainly on the application of machine learning and statistical data-driven approaches to the development of dialogue systems. However, it is important to be aware of previous achievements in dialogue technology and to consider to what extent they might be relevant to current research and development. Three main approaches to the development of dialogue systems are reviewed: rule-based systems that are handcrafted using best practice guidelines; statistical data-driven systems based on machine learning; and neural dialogue systems based on end-to-end learning. Evaluating the performance and usability of dialogue systems has become an important topic in its own right, and a variety of evaluation metrics and frameworks are described. Finally, a number of challenges for future research are considered, including: multimodality in dialogue systems, visual dialogue; data efficient dialogue model learning; using knowledge graphs; discourse and dialogue phenomena; hybrid approaches to dialogue systems development; dialogue with social robots and in the Internet of Things; and social and ethical issues.
Adversarial Learning for Invertible Steganography

Deep neural networks have revolutionised the research landscape of steganography. However, their potential has not been explored in invertible steganography, a special class of methods that permits the recovery of distorted objects due to steganographic perturbations to their pristine condition. In this paper, we revisit the regular-singular (RS) method and show that this elegant but obsolete invertible steganographic method can be reinvigorated and brought forwards to modern generation via neuralisation. Towards developing a renewed RS method, we introduce adversarial learning to capture the regularity of natural images automatically in contrast to handcrafted discrimination functions based on heuristic image prior. Specifically, we train generative adversarial networks (GANs) to predict bit-planes that have been used to carry hidden information. We then form a synthetic image and use it as a reference to provide guidance on data embedding and image recovery. Experimental results showed a significant improvement over the prior implementation of the RS method based on large-scale statistical evaluations.
Self-Loop Uncertainty: A Novel Pseudo-Label for Semi-supervised Medical Image Segmentation

Witnessing the success of deep learning neural networks in natural image processing, an increasing number of studies have been proposed to develop deep-learning-based frameworks for medical image segmentation. However, since the pixel-wise annotation of medical images is laborious and expensive, the amount of annotated data is usually deficient to well-train a neural network. In this paper, we propose a semi-supervised approach to train neural networks with limited labeled data and a large quantity of unlabeled images for medical image segmentation. A novel pseudo-label (namely self-loop uncertainty), generated by recurrently optimizing the neural network with a self-supervised task, is adopted as the ground-truth for the unlabeled images to augment the training set and boost the segmentation accuracy. The proposed self-loop uncertainty can be seen as an approximation of the uncertainty estimation yielded by ensembling multiple models with a significant reduction of inference time. Experimental results on two publicly available datasets demonstrate the effectiveness of our semi-supervised approach.
Artificial cognition: How experimental psychology can help generate explainable artificial intelligence

Artificial intelligence powered by deep neural networks has reached a level of complexity where it can be difficult or impossible to express how a model makes its decisions. This black-box problem is especially concerning when the model makes decisions with consequences for human well-being. In response, an emerging field called explainable artificial intelligence (XAI) aims to increase the interpretability, fairness, and transparency of machine learning. In this paper, we describe how cognitive psychologists can make contributions to XAI. The human mind is also a black box, and cognitive psychologists have over 150 years of experience modeling it through experimentation. We ought to translate the methods and rigor of cognitive psychology to the study of artificial black boxes in the service of explainability. We provide a review of XAI for psychologists, arguing that current methods possess a blind spot that can be complemented by the experimental cognitive tradition. We also provide a framework for research in XAI, highlight exemplary cases of experimentation within XAI inspired by psychological science, and provide a tutorial on experimenting with machines. We end by noting the advantages of an experimental approach and invite other psychologists to conduct research in this exciting new field.
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes

In this work we study the problem of using referential language to identify common objects in real-world 3D scenes. We focus on a challenging setup where the referred object belongs to a fine-grained object class and the underlying scene contains multiple object instances of that class. Due to the scarcity and unsuitability of existent 3D-oriented linguistic resources for this task, we first develop two large-scale and complementary visio-linguistic datasets: i) Sr3D, which contains 83.5 K template-based utterances leveraging spatial relations among fine-grained object classes to localize a referred object in a scene, and ii) Nr3D which contains 41.5K natural, free-form, utterances collected by deploying a 2-player object reference game in 3D scenes. Using utterances of either datasets, human listeners can recognize the referred object with high (>86%, 92% resp.) accuracy. By tapping on this data, we develop novel neural listeners that can comprehend object-centric natural language and identify the referred object directly in a 3D scene. Our key technical contribution is designing an approach for combining linguistic and geometric information (in the form of 3D point clouds) and creating multi-modal (3D) neural listeners . We also show that architectures which promote object-to-object communication via graph neural networks outperform less context-aware alternatives, and that fine-grained object classification is a bottleneck for language-assisted 3D object identification.
Soft errors in DNN accelerators: A comprehensive review

Deep learning tasks cover a broad range of domains and an even more extensive range of applications, from entertainment to extremely safety-critical fields. Thus, Deep Neural Network (DNN) algorithms are implemented on different systems, from small embedded devices to data centers. DNN accelerators have proven to be a key to efficiency, as they are even more efficient than CPUs. Therefore, they have become the major executing hardware for DNN algorithms. However, these accelerators are susceptible to several types of faults. Soft errors pose a particular threat because the high-level parallelism in these accelerators can propagate a single failure to multiple errors in the next levels until the model predictions' output is affected. This article presents a comprehensive review of the reliability of the DNN accelerators. The study begins by reviewing the widely assumed claim that DNNs are inherently tolerant to faults. Then, the available DNN accelerators are systematically classified into several categories. Each is individually analyzed; and the commonly used accelerators are compared in an attempt to answer the question, which accelerator is more reliable against transient faults? The concluding part of this review highlights the gray areas of the DNNs and predicts future research directions that will enhance its applicability. This study is expected to benefit researchers in the areas of deep learning, DNN accelerators, and reliability of this efficient paradigm.
Earthquake early warning in Aotearoa New Zealand: a survey of public perspectives to guide warning system development

Abstract Earthquake early warning (EEW) can be used to detect earthquakes and provide advanced notification of strong shaking, allowing pre-emptive actions to be taken that not only benefit infrastructure but reduce injuries and fatalities. Currently Aotearoa New Zealand does not have a nationwide EEW system, so a survey of the public was undertaken to understand whether EEW was considered useful and acceptable by the public, as well as perceptions of how and when such warnings should be communicated, before making an investment in such technology. We surveyed the public’s perspectives ( N = 3084) on the usefulness of EEW, preferred system attributes, and what people anticipated doing on receipt of a warning. We found strong support for EEW, for the purposes of being able to undertake actions to protect oneself and others (e.g. family, friends, and pets), and to mentally prepare for shaking. In terms of system attributes, respondents expressed a desire for being warned at a threshold of shaking intensity MM5–6. They suggested a preference for receiving a warning via mobile phone, supported by other channels. In addition to being warned about impending shaking, respondents wanted to receive messages that alerted them to other attributes of the earthquake (including the possibility of additional hazards such as tsunami), and what actions to take. People’s anticipated actions on receipt of a warning varied depending on the time available from the warning to arrival of shaking. People were more likely to undertake quicker and easier actions for shorter timeframes of &lt;10 s (e.g., stop, mentally prepare, take protective action), and more likely to move to a nearby safe area, help others, look for more information, or take safety actions as timeframes increased. Given the public endorsement for EEW, information from this survey can be used to guide future development in Aotearoa New Zealand and internationally with respect to system attributes, sources, channels and messages, in ways that promote effective action.
Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression

Precise estimation of the probabilistic structure of natural images plays an essential role in image compression. Despite the recent remarkable success of end-to-end optimized image compression, the latent codes are usually assumed to be fully statistically factorized in order to simplify entropy modeling. However, this assumption generally does not hold true and may hinder compression performance. Here we present context-based convolutional networks (CCNs) for efficient and effective entropy modeling. In particular, a 3D zigzag scanning order and a 3D code dividing technique are introduced to define proper coding contexts for parallel entropy decoding, both of which boil down to place translation-invariant binary masks on convolution filters of CCNs. We demonstrate the promise of CCNs for entropy modeling in both lossless and lossy image compression. For the former, we directly apply a CCN to the binarized representation of an image to compute the Bernoulli distribution of each code for entropy estimation. For the latter, the categorical distribution of each code is represented by a discretized mixture of Gaussian distributions, whose parameters are estimated by three CCNs. We then jointly optimize the CCN-based entropy model along with analysis and synthesis transforms for rate-distortion performance. Experiments on the Kodak and Tecnick datasets show that our methods powered by the proposed CCNs generally achieve comparable compression performance to the state-of-the-art while being much faster.