Reviewerly/exHarmony huggingface dataset:
POEM

Deep learning models achieve state-of-the-art performance in many applications, but their prediction decisions are difficult to explain. Various solutions exist in the area of explainable AI, for example to understand individual predictions or to approximate complex models using simpler interpretable ones. We contribute to this body of work with POEM: a tool that produces pattern-oriented explanations of image classification models. POEM explains models that learn hierarchies of concepts, such as Convolutional Neural Networks that detect shapes and objects in images. For example, POEM may identify a pattern of the form "if bed then bedroom", indicating that if an image contains a bed and the model pays attention to this region of the image during inference, then the model classifies the image as a bedroom. We present the modular design of POEM, followed by examples of POEM's use in model auditing and detecting errors in training data.
Optimized latent-code selection for explainable conditional text-to-image GANs

The task of text-to-image generation has achieved remarkable progress due to the advances in conditional generative adversarial networks (GANs).However, existing conditional text-to-image GANs approaches mostly concentrate on improving both image quality and semantic relevance but ignore the explainability of the model which plays a vital role in real-world applications.In this paper, we present a variety of techniques to take a deep look into the latent space and semantic space of a conditional text-to-image GANs model.We introduce pairwise linear interpolation of latent codes and 'linguistic' linear interpolation to study what the model has learned within the latent space and 'linguistic' embeddings.Subsequently, we extend linear interpolation to triangular interpolation conditioned on three corners to further analyze the model.After that, we build a Good/Bad data set containing unsuccessfully and successfully synthesized samples and corresponding latent codes for the image-quality research.Based on this data set, we propose a framework for finding good latent codes by utilizing a linear SVM.Experimental results on the recent DiverGAN generator trained on two benchmark data sets qualitatively prove the effectiveness of our presented techniques, with a better than 94% accuracy in predicting Good/Bad classes for latent vectors.The Good/Bad data set is publicly available at https://zenodo.org/record/5850224#.YeGMwP7MKUk.
Template-based Abstractive Microblog Opinion Summarization

Abstract We introduce the task of microblog opinion summarization (MOS) and share a dataset of 3100 gold-standard opinion summaries to facilitate research in this domain. The dataset contains summaries of tweets spanning a 2-year period and covers more topics than any other public Twitter summarization dataset. Summaries are abstractive in nature and have been created by journalists skilled in summarizing news articles following a template separating factual information (main story) from author opinions. Our method differs from previous work on generating gold-standard summaries from social media, which usually involves selecting representative posts and thus favors extractive summarization models. To showcase the dataset’s utility and challenges, we benchmark a range of abstractive and extractive state-of-the-art summarization models and achieve good performance, with the former outperforming the latter. We also show that fine-tuning is necessary to improve performance and investigate the benefits of using different sample sizes.
Hierarchical Motion Planning for Autonomous Vehicles in Unstructured Dynamic Environments

This letter presents a hierarchical motion planner for generating smooth and feasible trajectories for autonomous vehicles in unstructured environments with static and moving obstacles. The framework enables real-time computation by progressively shrinking the solution space. First, a graph searcher based on combined heuristic and partial motion planning is proposed for finding coarse trajectories in spatiotemporal space. To enable fast online planning, a time interval-based algorithm that considers obstacle prediction trajectories is proposed, which uses line segment intersection detection to check for collisions. Second, to practically smooth the coarse trajectory, a continuous optimizer is implemented in three layers, corresponding to the whole path, the near-future path and the speed profile. We use discrete points to represent the far-future path and parametric curves to represent the near-future path and the whole speed profile. The approach is validated in both simulations and real-world off-road environments based on representative scenarios, including the “wait and go” scenario. The experimental results show that the proposed method improves the success rate and travel efficiency while actively avoiding static and moving obstacles.
Adaptive Coverage Path Planning for Efficient Exploration of Unknown Environments

We present a method for solving the coverage problem with the objective of autonomously exploring an unknown environment under mission time constraints. Here, the robot is tasked with planning a path over a horizon such that the accumulated area swept out by its sensor footprint is maximized. Because this problem exhibits a diminishing returns property known as submodularity, we choose to formulate it as a tree-based sequential decision making process. This formulation allows us to evaluate the effects of the robot's actions on future world coverage states, while simultaneously accounting for traversability risk and the dynamic constraints of the robot. To quickly find near-optimal solutions, we propose an effective approximation to the coverage sensor model which adapts to the local environment. Our method was extensively tested across various complex environments and served as the local exploration algorithm for a competing entry in the DARPA Subterranean Challenge.
Learned Depth Estimation of 3D Imaging Radar for Indoor Mapping

3D imaging radar offers robust perception capability through visually demanding environments due to the unique penetrative and reflective properties of millimeter waves (mmWave). Current approaches for 3D perception with imaging radar require knowledge of environment geometry, accumulation of data from multiple frames for perception, or access to between-frame motion. Imaging radar presents an additional difficulty due to the complexity of its data representation. To address these issues, and make imaging radar easier to use for downstream robotics tasks, we propose a learning-based method that regresses radar measurements into cylindrical depth maps using LiDAR supervision. Due to the limitation of the regression formulation, directions where the radar beam could not reach will still generate a valid depth. To address this issue, our method additionally learns a 3D filter to remove those pixels. Experiments show that our system generates visually accurate depth estimation. Furthermore, we confirm the overall ability to generalize in the indoor scene using the estimated depth for probabilistic occupancy mapping with ground truth trajectory. The code and model will be released <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/rpl-cmu/learned-depth-imaging-radar.
High-Speed Accurate Robot Control using Learned Forward Kinodynamics and Non-linear Least Squares Optimization

Accurate control of robots at high speeds requires a control system that can take into account the kinodynamic interactions of the robot with the environment. Prior works on learning inverse kinodynamic (IKD) models of robots have shown success in capturing the complex kinodynamic effects. However, the types of control problems these approaches can be applied to are limited only to that of following pre-computed kinodynamically feasible trajectories. In this paper we present Optim-FKD, a new formulation for accurate, high-speed robot control that makes use of a learned forward kinodynamic (FKD) model and non-linear least squares optimization. Optim-FKD can be used for accurate, high speed control on any control task specifiable by a non-linear least squares objective. Optim-FKD can solve for control objectives such as path following and time-optimal control in real time, without needing access to pre-computed kinodynamically feasible trajectories. We empirically demonstrate these abilities of our approach through experiments on a scale one-tenth autonomous car. Our results show that Optim-FKD can follow desired trajectories more accurately and can find better solutions to optimal control problems than baseline approaches.
Detecting Vulnerabilities in Ethereum Smart Contracts with Deep Learning

Decentralized applications and smart contracts running on Ethereum have been widely applied in many fields, such as finance and logistics. On the one hand, the number of smart contracts surges. On the other hand, smart contracts often carry million dollars. Once there is a severe vulnerability in a smart contract, huge losses may occur. Detection tools like Oyente and Mythril were developed for vulnerabilities in smart contracts. However, many of these tools many detection tools do not use intrinsic features in the contract, resulting in ineffectiveness in the detection. In addition, many tools rely on symbolic execution and lack automation in the detection, resulting in inefficiency and in incapacity in vetting huge number of emerging smart contract on Ethereum. In our work, we are motivated to enhance the effectiveness and efficiency of vulnerability detection in contracts. First, we use opcodes as static features and use bigram to build the opcode feature space that is further optimized with Mtfidf. Second, We use deep learning algorithms namely, CNN, LSTM, CNN-BiLSTM, and ResNets, for the smart contract vulnerability detection. Extensive experimental results based on real-world smart contracts show that our methods are promising. Mtfidf significantly improves the detection performance. The best detection performance with ResNets reaches 82% in terms of Macro-F1.
A Proposal of Path Planning for Robots in Warehouses by Model Predictive Control without Using Global Paths

With the development of the e-commerce business, there are high expectations for the use of automated transport robots to improve the efficiency of warehouses. In this study, we propose a path planning method based on Model Predictive Control (MPC) that optimizes the performance of the entire warehouse for multiple robots. In general, when the MPC is used to execute path planning, global path planning such as Dijkstra algorithm are used together. This is because, if the MPC is used alone, the existence of a local optimal solution will cause a situation where the robot cannot move. However, in order to use the global route, it is necessary to prepare the precise map information of the warehouse in advance. In this study, we propose a path planning method that avoids deadlock without using global path by improving an objective function for the MPC. The effectiveness of the proposed method is verified in a numerical simulation.
Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft robotics-based surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geo-metric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demon-strate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.
Speaker-Independent Microphone Identification in Noisy Conditions

This work proposes a method for source device identification from speech recordings that applies neural-network-based denoising, to mitigate the impact of counter-forensics attacks using noise injection. The method is evaluated by comparing the impact of denoising on three state-of-the-art features for microphone classification, determining their discriminating power with and without denoising being applied. The proposed framework achieves a significant performance increase for noisy material, and more generally, validates the usefulness of applying denoising prior to device identification for noisy recordings.
AR-NeRF: Unsupervised Learning of Depth and Defocus Effects from Natural Images with Aperture Rendering Neural Radiance Fields

Fully unsupervised 3D representation learning has gained attention owing to its advantages in data collection. A successful approach involves a viewpoint-aware approach that learns an image distribution based on generative models (e.g., generative adversarial networks (GANs)) while generating various view images based on 3D-aware models (e.g., neural radiance fields (NeRFs)). However, they require images with various views for training, and consequently, their application to datasets with few or limited viewpoints remains a challenge. As a complementary approach, an aperture rendering GAN (AR-GAN) that employs a defocus cue was proposed. However, an AR-GAN is a CNN-based model and represents a defocus independently from a viewpoint change despite its high correlation, which is one of the reasons for its performance. As an alternative to an AR-GAN, we propose an aperture rendering NeRF (AR-NeRF), which can utilize viewpoint and defocus cues in a unified manner by representing both factors in a common ray-tracing framework. Moreover, to learn defocus-aware and defocus-independent representations in a disentangled manner, we propose aperture randomized training, for which we learn to generate images while randomizing the aperture size and latent codes independently. During our experiments, we applied AR-NeRF to various natural image datasets, including flower, bird, and face images, the results of which demonstrate the utility of AR-NeRF for un-supervised learning of the depth and defocus effects.
Wind Power Forecasting Based on Prophet Model

In order to improve the accuracy of wind power forecasting, a wind power time series forecasting model based on Prophet al-gorithm is established. The wind power data of a region from 2018 to 2020 are used for forecasting, and compared with the long short- term memory neural network forecasting model to explore the feasibility of the model in wind power forecasting.
Memristor Based Circuit Design for Liquid State Machine Verified with Temporal Classification

Spiking neural networks represent a transition from deep networks and tensor engines to more dynamic systems better suited for carrying out decisions based on temporal patterns within data. Furthermore, memristor based hardware for synaptic computation provides the potential for dramatic gains in terms of portability, power reduction, and throughput efficiency. In this paper we combine these ideas and present a memristor crossbar based implementation of a liquid state machine based on spiking neurons. The design is verified by demonstrating successful classification of MIDI (Musical Instrument Digital Interface) signals. Given the temporal nature of musical works, this dataset is well suited to show the effectiveness of the proposed hardware. The proposed circuit implementing this LSM is designed using SPICE to ensure accuracy at the device level, which aids in detailed analysis and circuit optimizations. Liquid layer activity is converted to state vectors using custom hardware that does not bottleneck throughput. Along with the proposed architecture, a neuron modeling procedure is described to match the software and hardware spiking neuron outputs, proving the feasibility of a full-scale implementation of the LSM system. Furthermore, various non-idealities are analyzed in the network, including misfiring neurons and noise associated with updating memristor conductance. We find that the proposed architecture is resilient to noise and fault-tolerant. In simulation, the performance of the memristor-based LSM is demonstrated on a musical genre classification experiment, where a recognition accuracy of 90% is achieved, compared to 91% accuracy in software. To the best of our knowledge this work represents the first end to end circuit implementation of an analog spiking neural network based reservoir computing system where memristors are responsible for all computation, with custom CMOS hardware for sampling the liquid layer. Finally, we tie the work together with an energy and timing analysis of the proposed system.
Human Action Recognition of Autonomous Mobile Robot Using Edge-AI

The development of autonomous mobile robots (AMRs) has brought with its requirements for intelligence and safety. Human action recognition (HAR) within AMR has become increasingly important because it provides interactive cognition between human and AMR. This study presents a full architecture for edge-artificial intelligence HAR (Edge-AI HAR) to allow AMR to detect human actions in real time. The architecture consists of three parts: a human detection and tracking network, a key frame extraction function, and a HAR network. The HAR network is a cascade of a DenseNet121 and a double-layer bidirectional long-short-term-memory (DLBiLSTM), in which the DenseNet121 is a pretrained model to extract spatial features from action key frames and the DLBiLSTM provides a deep two-directional LSTM inference to classify complicated time-series human actions. Edge-AI HAR undergoes two optimizations—ROS distributed computation and TensorRT structure optimization—to give a small model structure and high computational efficiency. Edge-AI HAR is demonstrated in two experiments using an AMR and is demonstrated to give an average precision of 97.58% for single action recognition and around 86% for continuous action recognition.
CloudBruno: A Low-Overhead Online Workload Prediction Framework for Cloud Computing

Accurate prediction of future incoming workloads to cloud applications, such as future user request count, is critical to proactive auto-scaling, and in general, critical to the cost-effectiveness of cloud deployments. However, designing a generic predictive framework that can accurately predict for any types of workloads is difficult, especially when the workload is dynamic and can change to a pattern that has not been observed in the training data sets. However, existing workload prediction solutions typically rely on complex machine learning models, which require comprehensive training data, making it difficult for them to handle dynamic workloads. Moreover, the training of existing workload prediction solutions are also expensive in terms of both time and computing resources. This paper presents a generic and low-cost online workload prediction framework, called Cloud Bruno, which combines the more accurate LSTM models with less expensive but fast SVM models to achieve high accuracy and low training overhead. When compared to existing predictors, CloudBruno had at least 8.8 % lower error than existing deep learning-based predictors for a highly-dynamic workload that does not have comprehensive training data (i.e, has changes unknown to training data). For workloads with comprehensive training data, Cloud Bruno's error was at most 2.5 % higher than optimized deep learning-based predictors. More importantly, Cloud Bruno can effectively execute on a free cloud CPU, allowing it to be used as an online workload predictor without additional cost.
Segmentation of ultrasound fetal image using spatial fuzzy C-Mean clustering method

Image segmentation is a vital and crucial part of image analysis and medical systems. This is the utmost challenging task as it decides the efficiency of the outcome of the image analysis. Ultrasound images play a significant role among other medical images. The automatic segmentation of these images becomes a factual challenge due to the speckle noise and the artifacts. The selection of the segmentation approach depends on the quality of the segmentation and the scale of information required. In this paper, the fetal ultrasound image is segmented using the Spatial Fuzzy C-Mean clustering method. The feature vectors are developed for each pixel of the fetal images used as inputs for the clustering method. The clustering methods segment the fetal image based on spatial information. An anisotropic Diffusion filter is used for image enhancement before the image segmentation. Experimental results indicate that the Spatial Fuzzy C-Means clustering method can be applied with promising results.
Selective Progressive Learning for Sparse Depth Completion

In this paper, we propose selective progressive learning (SPL) for sparse depth completion. Given an intermediate feature map selected from the input RGB and sparse depth image pairs by the valid and invalid mask, SPL sequentially infers attention maps along two separate dimensions (spatial attention and channel attention), and then the attention maps are used to extract features for depth regression. First, we design sparsity selection module (SSM) to simplify the task and segment it as depth prediction from sparse depth guided by corresponding RGB pixels and RGB to depth translation. Second, we propose a guided attention module (GAM) to translate features from SSM to attention maps by spatial and channel transformations. We use GAM as guidance for two pathways in the depth prediction network. Third, we combine two pathways by three cascade regressions with quarter-size, half-size and full-size of the input. We use multi-scale learning to maximize the advantage of cascade regressions, which utilizes down-sampling to get the ground truth of their corresponding size. The SPL network structure based on sparsity greatly reduces the amount of calculation, and SPL successfully bridges the modality gap between RGB and depth images for sparse depth completion. Experimental results show that SPL generates competitive dense depth images with low complexity and outperforms state-of-the-art ones in terms of both accuracy and robustness. Ablation experiments demonstrate the generalization capability and stability of SPL.