Reviewerly/exHarmony huggingface dataset:
Learning Illumination from Diverse Portraits

We present a learning-based technique for estimating high dynamic range (HDR), omnidirectional illumination from a single low dynamic range (LDR) portrait image captured under arbitrary indoor or outdoor lighting conditions. We train our model using portrait photos paired with their ground truth illumination. We generate a rich set of such photos by using a light stage to record the reflectance field and alpha matte of 70 diverse subjects in various expressions. We then relight the subjects using image-based relighting with a database of one million HDR lighting environments, compositing them onto paired high-resolution background imagery recorded during the lighting acquisition. We train the lighting estimation model using rendering-based loss functions and add a multi-scale adversarial loss to estimate plausible high frequency lighting detail. We show that our technique outperforms the state-of-the-art technique for portrait-based lighting estimation, and we also show that our method reliably handles the inherent ambiguity between overall lighting strength and surface albedo, recovering a similar scale of illumination for subjects with diverse skin tones. Our method allows virtual objects and digital characters to be added to a portrait photograph with consistent illumination. As our inference runs in real-time on a smartphone, we enable realistic rendering and compositing of virtual objects into live video for augmented reality.
Isotope identification using deep learning: An explanation

The exceptional performance of machine learning methods has led to their adaptation in many different domains. In the nuclear industry, it has been proposed that machine learning methods have the potential to revolutionize nuclear safety and radiation detection by leveraging that they can be used to augment human and device capabilities. While many applications focus on the accuracy of the learning algorithm’s prediction, it has been shown in practice that these algorithms are prone to learn characteristics that are not descriptive or relevant. Hence, this paper focuses on understanding the reasoning behind the classification using saliency methods. Visual representations of the network’s learned regions of interest are used to demonstrate whether domain-specific characteristics are being identified, which allows for the end-user to evaluate the performance based on domain knowledge. The results obtained show that focusing on a human-centered approach will ultimately enhance the transparency and trust of the deep learning algorithm’s decision.
Internet Traffic Classification with Federated Learning

As Internet traffic classification is a typical problem for ISPs or mobile carriers, there have been a lot of studies based on statistical packet header information, deep packet inspection, or machine learning. Due to recent advances in end-to-end encryption and dynamic port policies, machine or deep learning has been an essential key to improve the accuracy of packet classification. In addition, ISPs or mobile carriers should carefully deal with the privacy issue while collecting user packets for accounting or security. The recent development of distributed machine learning, called federated learning, collaboratively carries out machine learning jobs on the clients without uploading data to a central server. Although federated learning provides an on-device learning framework towards user privacy protection, its feasibility and performance of Internet traffic classification have not been fully examined. In this paper, we propose a federated-learning traffic classification protocol (FLIC), which can achieve an accuracy comparable to centralized deep learning for Internet application identification without privacy leakage. FLIC can classify new applications on-the-fly when a participant joins in learning with a new application, which has not been done in previous works. By implementing the prototype of FLIC clients and a server with TensorFlow, the clients gather packets, perform the on-device training job and exchange the training results with the FLIC server. In addition, we demonstrate that federated learning-based packet classification achieves an accuracy of 88% under non-independent and identically distributed (non-IID) traffic across clients. When a new application that can be classified dynamically as a client participates in learning was added, an accuracy of 92% was achieved.
Learning temporal data with a variational quantum recurrent neural network

We propose a method for learning temporal data using a parametrized quantum circuit. We use the circuit that has a similar structure as the recurrent neural network, which is one of the standard approaches employed for this type of machine learning task. Some of the qubits in the circuit are utilized for memorizing past data, while others are measured and initialized at each time step for obtaining predictions and encoding a new input datum. The proposed approach utilizes the tensor product structure to get nonlinearity with respect to the inputs. Fully controllable, ensemble quantum systems such as an NMR quantum computer are a suitable choice of an experimental platform for this proposal. We demonstrate its capability with simple numerical simulations, in which we test the proposed method for the task of predicting cosine and triangular waves and quantum spin dynamics. Finally, we analyze the dependency of its performance on the interaction strength among the qubits in numerical simulation and find that there is an appropriate range of the strength. This work provides a way to exploit complex quantum dynamics for learning temporal data.
BERT-GT: cross-sentence <i>n</i>-ary relation extraction with BERT and Graph Transformer

A biomedical relation statement is commonly expressed in multiple sentences and consists of many concepts, including gene, disease, chemical and mutation. To automatically extract information from biomedical literature, existing biomedical text-mining approaches typically formulate the problem as a cross-sentence n-ary relation-extraction task that detects relations among n entities across multiple sentences, and use either a graph neural network (GNN) with long short-term memory (LSTM) or an attention mechanism. Recently, Transformer has been shown to outperform LSTM on many natural language processing (NLP) tasks.In this work, we propose a novel architecture that combines Bidirectional Encoder Representations from Transformers with Graph Transformer (BERT-GT), through integrating a neighbor-attention mechanism into the BERT architecture. Unlike the original Transformer architecture, which utilizes the whole sentence(s) to calculate the attention of the current token, the neighbor-attention mechanism in our method calculates its attention utilizing only its neighbor tokens. Thus, each token can pay attention to its neighbor information with little noise. We show that this is critically important when the text is very long, as in cross-sentence or abstract-level relation-extraction tasks. Our benchmarking results show improvements of 5.44% and 3.89% in accuracy and F1-measure over the state-of-the-art on n-ary and chemical-protein relation datasets, suggesting BERT-GT is a robust approach that is applicable to other biomedical relation extraction tasks or datasets.the source code of BERT-GT will be made freely available at https://github.com/ncbi/bert_gt upon publication.Supplementary data are available at Bioinformatics online.
DNR

This paper presents a dynamic network rewiring (DNR) method to generate pruned deep neural network (DNN) models that are robust against adversarial attacks yet maintain high accuracy on clean images. In particular, the disclosed DNR method is based on a unified constrained optimization formulation using a hybrid loss function that merges ultra-high model compression with robust adversarial training. This training strategy dynamically adjusts inter-layer connectivity based on per-layer normalized momentum computed from the hybrid loss function. In contrast to existing robust pruning frameworks that require multiple training iterations, the proposed learning strategy achieves an overall target pruning ratio with only a single training iteration and can be tuned to support both irregular and structured channel pruning. To evaluate the merits of DNR, experiments were performed with two widely accepted models, namely VGG16 and ResNet-18, on CIFAR-10, CIFAR-100 as well as with VGG16 on Tiny-ImageNet. Compared to the baseline uncompressed models, DNR provides over 20x compression on all the datasets with no significant drop in either clean or adversarial classification accuracy. Moreover, our experiments show that DNR consistently finds compressed models with better clean and adversarial image classification performance than what is achievable through state-of-the-art alternatives. Our models and test codes are available at https://github.com/ksouvik52/DNR_ASP_DAC2021.
Blockchain-based mobile edge computing system

With the development of the Internet of Things (IoT), the number of mobile terminal devices is increasing rapidly. Due to high transmission delay and bandwidth limitation, computing power requirements for IoT devices are getting higher and higher. Recently, edge computing is an effective way to reduce system delay, and blockchain solves the security problem of edge computing. In this paper, a three-layer network model, named blockchain-based mobile edge computing system (BMEC), is proposed for clone block identification. Specifically, a neural network based clone block identification (NCBI) method is proposed to prevent clone block attacks. After that, the Prim algorithm is applied to BMEC to generate a weighted undirected graph minimum spanning tree that is composed of edge blocks. This can divide a main chain into several side chains to improve the transaction speed of blockchain. Finally, the blockchain is constructed based on the time slicing round-robin scheduling algorithm to control resources from edge servers and regulate edge devices' activities based on the predefined rules of priority, application type, and past behavior. Experimental results show that our clone block identification method can achieve block validation effectively in BMEC, and our construction of blockchain delay is lower than conventional edge computing methods.
Decoy-State Quantum Key Distribution Over a Long-Distance High-Loss Air-Water Channel

Atmospheric free space and fiber have been widely exploited as the channels for quantum communication, and have enabled intercontinent and intercity applications. An air-sea free-space channel, being capable of linking the satellite-based quantum resource and underwater vehicle, has now become the last piece of the puzzle in building a global quantum communication network. Here, we present an experimental demonstration of air-water decoy-state quantum key distribution against high loss, meanwhile keeping a low quantum bit error rate less than 2.5% for different distances. We are able to reach a long-distance quantum key distribution in water channel that can enable real-life air-sea quantum-communication tasks. The demonstrated distance, even in coastal water of Jerlov type III, is up to 30 m, about one-order improvement over the proof-in-principle demonstrations in the previous experiments, and the channel loss is equivalent to 345-m-long clean seawater of Jerlov type I, representing a key step forward to practical air-sea secure communication.
Evidential fully convolutional network for semantic segmentation

We propose a hybrid architecture composed of a fully convolutional network (FCN) and a Dempster-Shafer layer for image semantic segmentation. In the so-called evidential FCN (E-FCN), an encoder-decoder architecture first extracts pixel-wise feature maps from an input image. A Dempster-Shafer layer then computes mass functions at each pixel location based on distances to prototypes. Finally, a utility layer performs semantic segmentation from mass functions and allows for imprecise classification of ambiguous pixels and outliers. We propose an end-to-end learning strategy for jointly updating the network parameters, which can make use of soft (imprecise) labels. Experiments using three databases (Pascal VOC 2011, MIT-scene Parsing and SIFT Flow) show that the proposed combination improves the accuracy and calibration of semantic segmentation by assigning confusing pixels to multi-class sets.
A Safety-Aware Kinodynamic Architecture for Human-Robot Collaboration

The new paradigm of human-robot collaboration has led to the creation of shared work environments in which humans and robots work in close contact with each other. Consequently, the safety regulations have been updated addressing these new scenarios. The mere application of these regulations may lead to a very inefficient behavior of the robot. In order to preserve safety for the human operators and allow the robot to reach a desired configuration in a safe and efficient way, a two layers architecture for trajectory planning and scaling is proposed. The first layer calculates the nominal trajectory and continuously adapts it based on the human behavior. The second layer, which explicitly considers the safety regulations, scales the robot velocity and requests for a new trajectory if the robot speed drops. The proposed architecture is experimentally validated on a Pilz PRBT manipulator.
Predicting COVID-19 Based on Environmental Factors With Machine Learning

The coronavirus disease 2019 (COVID-19) has infected more than 50 million people in more than 100 countries, resulting in a major global impact. Many studies on the potential roles of environmental factors in the transmission of the novel COVID-19 have been published. However, the impact of environmental factors on COVID-19 remains controversial. Machine learning techniques have been used effectively in combating the COVID-19 epidemic. However, researches related to machine learning on weather conditions in spreading COVID-19 is generally lacking. Therefore, in this study, three machine learning models (Convolution Neural Network (CNN), ADtree Classifier and BayesNet) based on the confirmed cases and weather variables such as temperature, humidity, wind and precipitation are developed. This study aims to identify the best classification model to classify COVID-19 by using significant weather features chosen by Principle Component Analysis (PCA) feature selection method. The DS4C COVID-19 dataset is used to train and validate each machine learning model. Several data pre-processing tasks such as data cleaning and feature selection have been conducted on the raw dataset to ensure the quality of the training data. The performance of these machine learning algorithms is further rectified based on the selected features set by PCA. Each classifier is then optimized using different tuning parameters to achieve optimum values before comparing the output of the three classifiers against each other. The observational results have shown that the optimized CNN classifier with seven weather variables selected by PCA achieved the highest performance among all the techniques. The experimental results obtained show that the weather variables are more relevant in predicting the confirmed cases as compared to the other variables. Thus, from this result, it is evident that temperature, humidity, wind and precipitation are important features for predicting COVID-19 confirmed cases.
Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1

We present new speed records on the Armv8-A architecture for the latticebased schemes Dilithium, Kyber, and Saber. The core novelty in this paper is the combination of Montgomery multiplication and Barrett reduction resulting in “Barrett multiplication” which allows particularly efficient modular one-known-factor multiplication using the Armv8-A Neon vector instructions. These novel techniques combined with fast two-unknown-factor Montgomery multiplication, Barrett reduction sequences, and interleaved multi-stage butterflies result in significantly faster code. We also introduce “asymmetric multiplication” which is an improved technique for caching the results of the incomplete NTT, used e.g. for matrix-to-vector polynomial multiplication. Our implementations target the Arm Cortex-A72 CPU, on which our speed is 1.7× that of the state-of-the-art matrix-to-vector polynomial multiplication in kyber768 [Nguyen–Gaj 2021]. For Saber, NTTs are far superior to Toom–Cook multiplication on the Armv8-A architecture, outrunning the matrix-to-vector polynomial multiplication by 2.0×. On the Apple M1, our matrix-vector products run 2.1× and 1.9× faster for Kyber and Saber respectively.
Associative Knowledge Graph Using Fuzzy Clustering and Min-Max Normalization in Video Contents

Video content data have a variety of objects that could be associated with each other. Although content data contains similar objects or themes, their associations can become ambiguous. Accordingly, if associations between video content data are found in general association rules, their accuracy and confidence are low. Therefore, this study proposes the associative knowledge graph using fuzzy clustering and min-max normalization in video contents. With the use of the objects of video content, the proposed method finds clear and accurate associations between video content data and generates a knowledge graph. In the first step, the streaming video content data massively generated are collected, and objects in each image video are classified by an object detection algorithm. In the second step, normalization is executed in consideration of the different length of each video content and generate transaction data based on object frequency. In this way, it is possible to consider all the collected video content in the same condition. Additionally, it is possible to find unnecessary objects and significant objects in video content. Lastly, the degree of ambiguity is analyzed through fuzzy clustering using a probability that each object is involved in a group. Associations between fuzzy-clustered objects are extracted and an association knowledge graph is created. In this way, the accuracy and confidence of associations between different video content data are improved. As for performance, the association knowledge graph generated in the proposed method was better than a conventional association rule method in terms of the count of generated rules, support, and confidence.
Falx: Synthesis-Powered Visualization Authoring

Modern visualization tools aim to allow data analysts to easily create exploratory visualizations. When the input data layout conforms to the visualization design, users can easily specify visualizations by mapping data columns to visual channels of the design. However, when there is a mismatch between data layout and the design, users need to spend significant effort on data transformation.
Simulated quantum annealing as a simulator of nonequilibrium quantum dynamics

Simulated quantum annealing based on the path-integral Monte Carlo is one of the most common tools to simulate quantum annealing on classical hardware. Nevertheless, it is in principle highly nontrivial whether or not this classical algorithm can correctly reproduce the quantum dynamics of quantum annealing, particularly in the diabatic regime. We study this problem numerically through the generalized Kibble-Zurek mechanism of defect distribution in the simplest ferromagnetic one-dimensional transverse-field Ising model with and without coupling to the environment. We find that, in the absence of coupling to the environment, simulated quantum annealing correctly describes the annealing-time dependence of the average number of defects, but a detailed analysis of the defect distribution shows clear deviations from the theoretical prediction. When the system is open (coupled to the environment), the average number of defects does not follow the theoretical prediction but is qualitatively compatible with the numerical result by the infinite-time-evolving block decimation combined with the quasiadiabatic propagator path integral, which is valid in a very-short-time region. The distribution of defects in the open system turns out to be not far from the theoretical prediction. It is surprising that the classical stochastic dynamics of simulated quantum annealing ostensibly reproduces some aspects of the quantum dynamics. However, a serious problem is that it is hard to predict for which physical quantities in which system it is reliable. Those results suggest the necessity to exert a good amount of caution in using simulated quantum annealing to study the detailed quantitative aspects of the dynamics of quantum annealing.
Morphological classification of galaxies with deep learning: comparing 3-way and 4-way CNNs

ABSTRACT Classifying the morphologies of galaxies is an important step in understanding their physical properties and evolutionary histories. The advent of large-scale surveys has hastened the need to develop techniques for automated morphological classification. We train and test several convolutional neural network (CNN) architectures to classify the morphologies of galaxies in both a 3-class (elliptical, lenticular, and spiral) and a 4-class (+irregular/miscellaneous) schema with a data set of 14 034 visually classified SDSS images. We develop a new CNN architecture that outperforms existing models in both 3-way and 4-way classifications, with overall classification accuracies of 83 and 81 per cent, respectively. We also compare the accuracies of 2-way/binary classifications between all four classes, showing that ellipticals and spirals are most easily distinguished (&amp;gt;98 per cent accuracy), while spirals and irregulars are hardest to differentiate (78 per cent accuracy). Through an analysis of all classified samples, we find tentative evidence that misclassifications are physically meaningful, with lenticulars misclassified as ellipticals tending to be more massive, among other trends. We further combine our binary CNN classifiers to perform a hierarchical classification of samples, obtaining comparable accuracies (81 per cent) to the direct 3-class CNN, but considerably worse accuracies in the 4-way case (65 per cent). As an additional verification, we apply our networks to a small sample of Galaxy Zoo images, obtaining accuracies of 92, 82, and 77 per cent for the binary, 3-way, and 4-way classifications, respectively.
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning

It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps). In this challenge, no additional image-caption training data, other than COCO Captions, is allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pre-training (VIVO) that performs pre-training in the absence of caption annotations. By breaking the dependency of paired image-caption training data in VLP, VIVO can leverage large amounts of paired image-tag data to learn a visual vocabulary. This is done by pre-training a multi-layer Transformer model that learns to align image-level tags with their corresponding image region features. To address the unordered nature of image tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct pre-training. We validate the effectiveness of VIVO by fine-tuning the pre-trained model for image captioning. In addition, we perform an analysis of the visual-text alignment inferred by our model. The results show that our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects. Our single model has achieved new state-of-the-art results on nocaps and surpassed the human CIDEr score.