Reviewerly/exHarmony huggingface dataset:
Auto-encoding and Distilling Scene Graphs for Image Captioning

We propose scene graph auto-encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inferences in discourse. For example, when we see the relation "a person on a bike", it is natural to replace "on" with "ride" and infer "a person riding a bike on a road" even when the "road" is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models reason as we humans and generate more descriptive captions. Specifically, we use the scene graph-a directed graph ( G) where an object node is connected by adjective nodes and relationship nodes-to represent the complex structural layout of both image ( I) and sentence ( S). In the language domain, we use SGAE to learn a dictionary set ( D) that helps reconstruct sentences in the S→ GS → D → S auto-encoding pipeline, where D encodes the desired language prior and the decoder learns to caption from such a prior; in the vision-language domain, we share D in the I→ GI → D → S pipeline and distill the knowledge of the language decoder of the auto-encoder to that of the encoder-decoder based image captioner to transfer the language inductive bias. In this way, the shared D provides hidden embeddings about descriptive collocations to the encoder-decoder and the distillation strategy teaches the encoder-decoder to transform these embeddings to human-like captions as the auto-encoder. Thanks to the scene graph representation, the shared dictionary set, and the Knowledge Distillation strategy, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, where our SGAE-based single-model achieves a new state-of-the-art 129.6 CIDEr-D on the Karpathy split, and a competitive 126.6 CIDEr-D (c40) on the official server, which is even comparable to other ensemble models. Furthermore, we validate the transferability of SGAE on two more challenging settings: transferring inductive bias from other language corpora and unpaired image captioning. Once again, the results of both settings confirm the superiority of SGAE. The code is released in https://github.com/yangxuntu/SGAE.
Deep Dual Relation Modeling for Egocentric Interaction Recognition

Egocentric interaction recognition aims to recognize the camera wearer's interactions with the interactor who faces the camera wearer in egocentric videos. In such a human-human interaction analysis problem, it is crucial to explore the relations between the camera wearer and the interactor. However, most existing works directly model the interactions as a whole and lack modeling the relations between the two interacting persons. To exploit the strong relations for egocentric interaction recognition, we introduce a dual relation modeling framework which learns to model the relations between the camera wearer and the interactor based on the individual action representations of the two persons. Specifically, we develop a novel interactive LSTM module, the key component of our framework, to explicitly model the relations between the two interacting persons based on their individual action representations, which are collaboratively learned with an interactor attention module and a global-local motion module. Experimental results on three egocentric interaction datasets show the effectiveness of our method and advantage over state-of-the-arts.
Finding the ground state of spin Hamiltonians with reinforcement learning

Reinforcement learning (RL) has become a proven method for optimizing a procedure for which success has been defined, but the specific actions needed to achieve it have not. Using a method we call ‘controlled online optimization learning’ (COOL), we apply the so-called ‘black box’ method of RL to simulated annealing (SA), demonstrating that an RL agent based on proximal policy optimization can, through experience alone, arrive at a temperature schedule that surpasses the performance of standard heuristic temperature schedules for two classes of Hamiltonians. When the system is initialized at a cool temperature, the RL agent learns to heat the system to ‘melt’ it and then slowly cool it in an effort to anneal to the ground state; if the system is initialized at a high temperature, the algorithm immediately cools the system. We investigate the performance of our RL-driven SA agent in generalizing to all Hamiltonians of a specific class. When trained on random Hamiltonians of nearest-neighbour spin glasses, the RL agent is able to control the SA process for other Hamiltonians, reaching the ground state with a higher probability than a simple linear annealing schedule. Furthermore, the scaling performance (with respect to system size) of the RL approach is far more favourable, achieving a performance improvement of almost two orders of magnitude on L = 142 systems. We demonstrate the robustness of the RL approach when the system operates in a ‘destructive observation’ mode, an allusion to a quantum system where measurements destroy the state of the system. The success of the RL agent could have far-reaching impacts, from classical optimization, to quantum annealing and to the simulation of physical systems. Reinforcement learning has become a popular method in various domains, for problems where an agent must learn what actions must be taken to reach a particular goal. An interesting example where the technique can be applied is simulated annealing in condensed matter physics, where a procedure is determined for slowly cooling a complex system to its ground state. A reinforcement learning approach has been developed that can learn a temperature scheduling protocol to find the ground state of spin glasses, magnetic systems with strong spin–spin interactions between neighbouring atoms.
A Two-Stage Privacy Preservation and Secure Peer-to-Peer Energy Trading Model Using Blockchain and Cloud-Based Aggregator

Local energy trading has attracted the attention of many researchers as a result of its promising benefits. These benefits include minimizing gas emission, reducing power shortage, and establishing a competitive energy market. However, the energy trading between several prosumers causes trust, security, and privacy challenges in energy systems. On the other hand, a single point of failure and an increase in overall system cost occur when the energy system is managed using a centralized model. Therefore, to tackle the mentioned issues, this work proposes a two-layered secure Peer-to-Peer (P2P) energy trading model based on blockchain. The proposed model has two layers: authentication, and secure energy trading. In the authentication layer, in order to protect the proposed model from impersonation attacks, a mutual authentication process is implemented. In the energy trading layer, a new consensus mechanism is proposed to minimize the number of malicious validators. Afterwards, an incentive-punishment algorithm is introduced to motivate energy prosumers to contribute more energy in the model. Next, a dynamic contract theory based on supply-demand ratio pricing scheme is proposed. The purpose of the proposed pricing scheme is to solve the issues associated with the existing pricing schemes. It also preserves the privacy of the actual energy consumption behavior of the trading participants. Furthermore, a consensus mechanism validators’ selection model is proposed. The aim of the proposed work is to have an efficient and secure P2P energy trading platform. Simulations are executed to show the performance of the proposed model in terms of communication and computational costs, reputation, energy contributed, reward, and prices. The results for the authentication process show 7.45 ms computational cost and 1152 bits communication cost, which are better than the existing works. In the consensus process, 66.67% of the validators are selected to conduct the consensus for every transaction. This selection efficiently improves the consensus process and minimizes the number of malicious validators. In the proposed model, the increase in reward is observed for increased energy contribution, decreased non-malicious transactions and adjustment of energy consumption. The proposed model shows a satisfactory performance in terms of trust, security, and privacy.
Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds

There has been a strong demand for algorithms that can execute machine learning as faster as possible and the speed of deep learning has accelerated by 30 times only in the past two years. Distributed deep learning using the large mini-batch is a key technology to address the demand and is a great challenge as it is difficult to achieve high scalability on large clusters without compromising accuracy. In this paper, we introduce optimization methods which we applied to this challenge. We achieved the training time of 74.7 seconds using 2,048 GPUs on ABCI cluster applying these methods. The training throughput is over 1.73 million images/sec and the top-1 validation accuracy is 75.08%.
C-loss based extreme learning machine for estimating power of small-scale turbojet engine

As a high-efficiency training method for single layer feedforward neural networks, extreme learning machine (ELM) has drawn much interest recently, but its robustness is not good due to the adoption of the square loss function. Hence, the convex loss function in ELM is replaced with a nonconvex loss function, i.e., the C-loss function, so a novel algorithm, called C-loss based ELM (CELM), is proposed in this paper. According to the experimental results on a toy example and two benchmark data sets, CELM performs better than the other algorithms with respect to the generalization performance. To be more important, when CELM is used to estimate power of small-scale turbojet engine, it also dominates the other algorithms, which makes the development of the potential control structure, viz., the direct power control, for the unmanned aerial vehicles more feasible.
FSSPOTTER: Spotting Face-Swapped Video by Spatial and Temporal Clues

Recent advances in face generation and manipulation have enabled the creation of sophisticated face-swapped videos, also known as DeepFakes, which brings great potential threats to our society. Hence, it is crucial to develop effective approaches to distinguish them. Currently, face-swapped videos produced by existing methods are prone to exhibit some subtle spatial and temporal manipulated traces, which can be utilized as distinctive clues for face-swapped video detection. In this paper, we propose a unified framework, named FSSpotter, to explore rich spatial and temporal information in the video simultaneously. It consists of a Spatial Feature Extractor (SFE), which aims to discover spatial evidences within a single frame, and a Temporal Feature Aggregator (TFA), which is responsible for capturing temporal inconsistencies between frames. Moreover, a novel data processing strategy is adopted to highlight the inconsistencies of forged face with its surrounding regions. The evaluations on Deepfakes of FaceForensics++, DeepfakeTIMIT, UADFV and Celeb-DF datasets demonstrate that the proposed approach achieves better or comparable performance on AUC scores.
Local Variational Feature-Based Similarity Models for Recommending Top- <i>N</i> New Items

The top- N recommendation problem has been studied extensively. Item-based collaborative filtering recommendation algorithms show promising results for the problem. They predict a user’s preferences by estimating similarities between a target and user-rated items. Top- N recommendation remains a challenging task in scenarios where there is a lack of preference history for new items. Feature-based Similarity Models (FSMs) address this particular problem by extending item-based collaborative filtering by estimating similarity functions of item features. The quality of the estimated similarity function determines the accuracy of the recommendation. However, existing FSMs only estimate global similarity functions; i.e., they estimate using preference information across all users. Moreover, the estimated similarity functions are linear ; hence, they may fail to capture the complex structure underlying item features. In this article, we propose to improve FSMs by estimating local similarity functions, where each function is estimated for a subset of like-minded users. To capture global preference patterns, we extend the global similarity function from linear to nonlinear, based on the effectiveness of variational autoencoders. We propose a Bayesian generative model, called the Local Variational Feature-based Similarity Model, to encapsulate local and global similarity functions. We present a variational Expectation Minimization algorithm for efficient approximate inference. Extensive experiments on a large number of real-world datasets demonstrate the effectiveness of our proposed model.
Self Domain Adapted Network

Domain shift is a major problem for deploying deep networks in clinical practice. Network performance drops significantly with (target) images obtained differently than its (source) training data. Due to a lack of target label data, most work has focused on unsupervised domain adaptation (UDA). Current UDA methods need both source and target data to train models which perform image translation (harmonization) or learn domain-invariant features. However, training a model for each target domain is time consuming and computationally expensive, even infeasible when target domain data are scarce or source data are unavailable due to data privacy. In this paper, we propose a novel self domain adapted network (SDA-Net) that can rapidly adapt itself to a single test subject at the testing stage, without using extra data or training a UDA model. The SDA-Net consists of three parts: adaptors, task model, and auto-encoders. The latter two are pre-trained offline on labeled source images. The task model performs tasks like synthesis, segmentation, or classification, which may suffer from the domain shift problem. At the testing stage, the adaptors are trained to transform the input test image and features to reduce the domain shift as measured by the auto-encoders, and thus perform domain adaptation. We validated our method on retinal layer segmentation from different OCT scanners and T1 to T2 synthesis with T1 from different MRI scanners and with different imaging parameters. Results show that our SDA-Net, with a single test subject and a short amount of time for self adaptation at the testing stage, can achieve significant improvements.
Optimal Batch Distributed Asynchronous Multisensor Fusion With Feedback

Asynchronous multisensor systems have been widely equipped on various host platforms to meet the requirements of modern navigation campaigns. However, the existing asynchronous fusion algorithms are not efficient in many challenged environments, due to the limitations of communication condition, out-of-sequence observation, and computation affordance. To solve these problems, this paper proposes a batch state asynchronous fusion algorithm with feedback for distributed architecture systems. It is optimal in the sense of minimum mean-squared estimation and its fusion solution is rigorously derived by fully considering the correlations of sequential/cross channels and one-step prediction information. The final fusion solution is straightforward expressed with the combinations of multiple local Kalman filtering estimates outputs during one fusion period. Moreover, no additional interpolation, extrapolation or synchronization procedures are required. Furthermore, to alleviate the computational burden of our proposed algorithm, its approximation form is also given by replacing the weighting matrix by the weighting scalar. Simulations are carried out to illustrate the efficiency and the performance of the proposed algorithm.
Wide Weighted Attention Multi-Scale Network for Accurate MR Image Super-Resolution

High-quality magnetic resonance (MR) images afford more detailed information for reliable diagnoses and quantitative image analyses. Given low-resolution (LR) images, the deep convolutional neural network (CNN) has shown its promising ability for image super-resolution (SR). The LR MR images usually share some visual characteristics: structural textures of different sizes, edges with high correlation, and less informative background. However, multi-scale structural features are informative for image reconstruction, while the background is more smooth. Most previous CNN-based SR methods use a single receptive field and equally treat the spatial pixels (including the background). It neglects to sense the entire space and get diversified features from the input, which is critical for high-quality MR image SR. We propose a wide weighted attention multi-scale network ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{W}^{2}$ </tex-math></inline-formula> AMSN) for accurate MR image SR to address these problems. On the one hand, the features of varying sizes can be extracted by the wide multi-scale branches. On the other hand, we design a non-reduction attention mechanism to recalibrate feature responses adaptively. Such attention preserves continuous cross-channel interaction and focuses on more informative regions. Meanwhile, the learnable weighted factors fuse extracted features selectively. The encapsulated wide weighted attention multi-scale block ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{W}^{2}$ </tex-math></inline-formula> AMSB) is integrated through a recurrent framework and global attention mechanism. Extensive experiments and diversified ablation studies show the effectiveness of our proposed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{W}^{2}$ </tex-math></inline-formula> AMSN, which surpasses state-of-the-art methods on most popular MR image SR benchmarks quantitatively and qualitatively. And our method still offers superior accuracy and adaptability on real MR images.
An effective blockchain-based, decentralized application for smart building system management

A smart building system can help us to optimize the capability of all the equipment and systems in a building. Blockchain was first introduced by Bitcoin in 2009 and developers all around the world have been trying to apply blockchain in different areas, like finance services, credit and ownership management, resource sharing, investment management, Internet of things (IoT), and so on. Ethereum is a blockchain 2.0 platform that allows developers to build a decentralized application (DApp) without building a new blockchain from scratch. IoT is the technology to embed all physical devices with sensors and chips to provide automation via machine-to-machine communication. Blynk is a platform that provides iOS and Android applications for users and developers to collect data from microcontroller. Our goal in this chapter is to build a blockchain-based, DApp named Uranus for smart building system management with private Ethereum blockchain, Raspberry Pi (RPi), Blynk platform, and sensors (e.g., DHT11). The blockchain technology in Uranus provides three benefits: (1) it is tamper-resistant, (2) it has no single point of failure, and (3) it is trusted and auditable. The system is a proof-of-concept (PoC) system prototype to simulate smart building applications. It collects real-time building temperature and humidity through a DHT11 sensor via RPi. The sensor data is updated to the Blynk app and stored on the smart contract deployed on the private Ethereum blockchain. If the real-time temperature or humidity value exceeds the threshold value set by the users, red or green LEDs will be turned on as warnings and then an air conditioner or dehumidifier will be turned on. We build a PoC smart building system to verify its feasibility and demonstrate its performance based on a simulated testbed.
A Crowd Analysis Framework for Detecting Violence Scenes

This work examines violence detection in video scenes of crowds and proposes a crowd violence detection framework based on a 3D convolutional deep learning architecture, the 3D-ResNet model with 50 layers. The proposed framework is evaluated on the Violent Flows dataset against several state-of-the-art approaches and achieves higher accuracy values in almost all cases, while also performing the violence detection activities in (near) real-time.
Generative Adversarial Registration for Improved Conditional Deformable Templates

Deformable templates are essential to large-scale medical image registration, segmentation, and population analysis. Current conventional and deep network-based methods for template construction use only regularized registration objectives and often yield templates with blurry and/or anatomically implausible appearance, confounding downstream biomedical interpretation. We reformulate deformable registration and conditional template estimation as an adversarial game wherein we encourage realism in the moved templates with a generative adversarial registration framework conditioned on flexible image covariates. The resulting templates exhibit significant gain in specificity to attributes such as age and disease, better fit underlying group-wise spatiotemporal trends, and achieve improved sharpness and centrality. These improvements enable more accurate population modeling with diverse covariates for standardized downstream analyses and easier anatomical delineation for structures of interest.
3D Trajectory Planning for UAV-based Search Missions: An Integrated Assessment and Search Planning Approach

The ability to efficiently plan and execute search missions in challenging and complex environments during natural and man-made disasters is imperative. In many emergency situations, precise navigation between obstacles and time-efficient searching around 3D structures is essential for finding survivors. In this work we propose an integrated assessment and search planning approach which allows an autonomous UAV (unmanned aerial vehicle) agent to plan and execute collision-free search trajectories in 3D environments. More specifically, the proposed search-planning framework aims to integrate and automate the first two phases (i.e., the assessment phase and the search phase) of a traditional search-and-rescue (SAR) mission. In the first stage, termed assessment-planning we aim to find a high-level assessment plan which the UAV agent can execute in order to visit a set of points of interest. The generated plan of this stage guides the UAV to fly over the objects of interest thus providing a first assessment of the situation at hand. In the second stage, termed search-planning, the UAV trajectory is further fine-tuned to allow the UAV to search in 3D (i.e., across all faces) the objects of interest for survivors. The performance of the proposed approach is demonstrated through extensive simulation analysis.
Visualizing expert motion for guidance in a VR ski simulator

While humans are quite good at copying motions from others, it is difficult to do so in a dynamic sport such as skiing. Hence, we propose a virtual reality ski training system, which visualizes prerecorded expert motion in different ways and enables users to learn by copying. The system is based on a commercial indoor ski simulator, a VR headset, and two VR trackers to capture the ski's motion. Users can control their skis on the virtual ski slope and improve their skills by following a digital avatar of the expert skier replayed in front of them. We investigate 3 types of visualizations for training: Graphs to visualize the angle of feet compared to the expert, periodic copies of the expert's pose to show the spatial and temporal motion of the key movements, and a more minimal ribbon-trace of the leading skier to point out the optimized trajectory.