Reviewerly/exHarmony huggingface dataset:
Adversarially Trained Actor Critic for Offline Reinforcement Learning

We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism. ATAC is designed as a two-player Stackelberg game: A policy actor competes against an adversarially trained value critic, who finds data-consistent scenarios where the actor is inferior to the data-collection behavior policy. We prove that, when the actor attains no regret in the two-player game, running ATAC produces a policy that provably 1) outperforms the behavior policy over a wide range of hyperparameters that control the degree of pessimism, and 2) competes with the best policy covered by data with appropriately chosen hyperparameters. Compared with existing works, notably our framework offers both theoretical guarantees for general function approximation and a deep RL implementation scalable to complex environments and large datasets. In the D4RL benchmark, ATAC consistently outperforms state-of-the-art offline RL algorithms on a range of continuous control tasks.
Learning Long-Term Reward Redistribution via Randomized Return Decomposition

Many practical applications of reinforcement learning require agents to learn from sparse and delayed rewards. It challenges the ability of agents to attribute their actions to future outcomes. In this paper, we consider the problem formulation of episodic reinforcement learning with trajectory feedback. It refers to an extreme delay of reward signals, in which the agent can only obtain one reward signal at the end of each trajectory. A popular paradigm for this problem setting is learning with a designed auxiliary dense reward function, namely proxy reward, instead of sparse environmental signals. Based on this framework, this paper proposes a novel reward redistribution algorithm, randomized return decomposition (RRD), to learn a proxy reward function for episodic reinforcement learning. We establish a surrogate problem by Monte-Carlo sampling that scales up least-squares-based reward redistribution to long-horizon problems. We analyze our surrogate loss function by connection with existing methods in the literature, which illustrates the algorithmic properties of our approach. In experiments, we extensively evaluate our proposed method on a variety of benchmark tasks with episodic rewards and demonstrate substantial improvement over baseline algorithms.
UrbanFly: Uncertainty-Aware Planning for Navigation Amongst High-Rises with Monocular Visual-Inertial SLAM Maps

We present UrbanFly: an uncertainty-aware real-time planning framework for quadrotor navigation in urban high-rise environments. A core aspect of UrbanFly is its ability to robustly plan directly on the sparse point clouds generated by a Monocular Visual Inertial SLAM (VINS) backend. It achieves this by using the sparse point clouds to build an uncertainty-integrated cuboid representation of the environment through a data-driven monocular plane segmentation network. Our chosen world model provides faster distance queries than the more common voxel-grid representation, and UrbanFly leverages this capability in two different ways leading to two trajectory optimizers. The first optimizer uses a gradient-free cross-entropy method to compute trajectories that minimize collision probability and smoothness cost. Our second optimizer is a simplified version of the first and uses a sequential convex programming optimizer initialized based on probabilistic safety estimates on a set of randomly drawn trajectories. Both our trajectory optimizers are made computationally tractable and independent of the nature of underlying uncertainty by embedding the distribution of collision violations in Reproducing Kernel Hilbert Space. Empowered by the algorithmic innovation, UrbanFly outperforms competing baselines in metrics such as collision rate, trajectory length, etc., on a high-fidelity AirSim simulator augmented with synthetic and real-world dataset scenes.
Self-Imitation Learning from Demonstrations

Despite the numerous breakthroughs achieved with Reinforcement Learning (RL), solving environments with sparse rewards remains a challenging task that requires sophisticated exploration. Learning from Demonstrations (LfD) remedies this issue by guiding the agent's exploration towards states experienced by an expert. Naturally, the benefits of this approach hinge on the quality of demonstrations, which are rarely optimal in realistic scenarios. Modern LfD algorithms require meticulous tuning of hyperparameters that control the influence of demonstrations and, as we show in the paper, struggle with learning from suboptimal demonstrations. To address these issues, we extend Self-Imitation Learning (SIL), a recent RL algorithm that exploits the agent's past good experience, to the LfD setup by initializing its replay buffer with demonstrations. We denote our algorithm as SIL from Demonstrations (SILfD). We empirically show that SILfD can learn from demonstrations that are noisy or far from optimal and can automatically adjust the influence of demonstrations throughout the training without additional hyperparameters or handcrafted schedules. We also find SILfD superior to the existing state-of-the-art LfD algorithms in sparse environments, especially when demonstrations are highly suboptimal.
Collapse and Revival of n-Photon Coherent States and n-Photon Squeezed Coherent States

We introduce the set of n-photon coherent states and the set of n-photon squeezed coherent states and study their collapse and revival and compare their behavior with the collapse and revival of the corresponding zero-photon coherent states and zero-photon squeezed coherent states, respectively. The set of n-photon squeezed coherent states (n = 0, 1, 2, ···) forms a basis of the Hilbert state of photons and may be of some interest. We notice that the presence of a few extra photons in the n-photon states as compared to the corresponding zero-photon states makes a large difference in the behavior of their collapse and revival. This indicates the large effect that a few extra photons make in the n-photon states as compared to the corresponding zero-photon states in their collapse and revival.
Algorithm and Circuit of Nesting Doubled Qubits

Copying the quantum states is contradictory to classical information processing since the fundamental difference between classical and quantum information is that while classical information can be copied perfectly, quantum information cannot. However, this statement does not rule out the risk of building a device that can reproduce a set of quantum states. This paper investigates the naturally arising question of how well or under what conditions one can copy and measure an arbitrary quantum superposition of states. The CNOT and XOR operation-based quantum circuit is presented that exhibits entanglement of states and allows for measuring the doubled qubits.
Improved Pure Exploration in Linear Bandits with No-Regret Learning

We study the best arm identification problem in linear multi-armed bandits (LMAB) in the fixed confidence setting ; this is also the problem of optimizing an unknown linear function over a discrete domain with noisy, zeroth-order access. We propose an explicitly implementable and provably order-optimal sample-complexity algorithm for best arm identification. Existing approaches that achieve optimal sample complexity assume either access to a nontrivial minimax optimization oracle (e.g. RAGE and Lazy-TS) or knowledge of an upper-bound on the norm of the unknown parameter vector(e.g. LinGame). Our algorithm, which we call the Phased Elimination Linear Exploration Game (PELEG), maintains a high-probability confidence ellipsoid containing the unknown vector, and uses it to eliminate suboptimal arms in phases, where a minimax problem is essentially solved in each phase using two-player low regret learners. PELEG does not require to know a bound on norm of the unknown vector, and is asymptotically-optimal, settling an open problem. We show that the sample complexity of PELEG matches, up to order and in a non-asymptotic sense, an instance-dependent universal lower bound for linear bandits. PELEG is thus the first algorithm to achieve both order-optimal sample complexity and explicit implementability for this setting. We also provide numerical results for the proposed algorithm consistent with its theoretical guarantees.
Historic Overview and Future Outlook of Blockchain Interoperability

Abstract For three decades, the Bitcoin network processes transactions collectively worth billions in a fully decentralized and trustless way. Its introduction as a way for disintermediating financial institutions came at a time of rising dismay against the establishment due to the financial collapse of 2008. Bitcoin’s rising popularity gave birth to the realization that its underlying technologies could be utilized for other use-cases besides money. This gave rise to imitators, iterators and ultimately a diverse ecosystem of protocols and applications. In most cases, those aforementioned protocols employed technologically proprietary approaches even when aiming to solve similar problems. Today, those separate networks stand as monolithic structures, with no knowledge of information that might exist on another. As such they are hostages to their non-interoperable nature and bound to hardcoded decisions. Their attempts at change often result in divided communities and further balkanization. This dissolution threatens the integrity of the decentralized space, as desolate systems are susceptible to manipulation. Some believe that the future of the wider decentralised ecosystem will rely on a Web 3.0 internet-like infrastructure that will allow for seamless integration and the free exchange of information between systems. In this chapter, we will explore the source of the limitations of blockchain systems, provide a historical overview of current and past interoperable approaches and discuss present and future solution design.
Enhancing Anomaly Detection Performance and Acceleration

Automation of visual inspection is a critical aspect in industrial fields. Recently, research on anomaly detection using neural networks has been gaining increasing attention. In particular, approaches that use a pre-trained convolutional neural network have exhibited high performance. In this study, we focused on PatchCore, which is a high-performance model, and further improved it using two high-resolution images to accurately detect small anomalies. However, the extracted features (memory bank) consume a large amount of memory and storage; the memory bank is compressed by k-means clustering. Moreover, the inference time was reduced by an approximate nearest-neighbor search using an inverted index. Our method achieved an image-level AUROC of 0.994 on the MVTec anomaly detection dataset. In addition, a pixel-level AUROC of 0.984 was achieved, which is better than that of PatchCore. Furthermore, the compression time was reduced by more than 97% by clustering the memory bank using k-means while maintaining the performance.
Universal Spam Detection using Transfer Learning of BERT Model

Deep learning transformer models become important by training on text data based on self-attention mechanisms. This manuscript demonstrated a novel universal spam detection model using pre-trained Google's Bidirectional Encoder Representations from Transformers (BERT) base uncased models with four datasets by efficiently classifying ham or spam emails in real-time scenarios. Different methods for Enron, Spamassain, Lingspam, and Spamtext message classification datasets, were used to train models individually in which a single model was obtained with acceptable performance on four datasets. The Universal Spam Detection Model (USDM) was trained with four datasets and leveraged hyperparameters from each model. The combined model was finetuned with the same hyperparameters from these four models separately. When each model using its corresponding dataset, an F1-score is at and above 0.9 in individual models. An overall accuracy reached 97%, with an F1 score of 0.96. Research results and implications were discussed.
Safe Control of Arbitrary Nonlinear Systems using Dynamic Extension

Safe control for control-affine systems has been extensively studied. However, due to the complexity of system dynamics, it is challenging and time-consuming to apply these methods directly to non-control-affine systems, which cover a large group of dynamic systems, such as UAVs and systems with data-driven Neural Network Dynamic Models (NNDMs). Although all dynamic systems can be written in control-affine forms through dynamic extension, it remains unclear how to optimally design a computationally efficient algorithm to safely control the extended system. This paper addresses this challenge by proposing an optimal approach to synthesize safe control for the extended system under the framework of energy-function-based safe control. The proposed method first extends the energy function and then performs hyperparameter optimization to maximize performance while guaranteeing safety. It has been theoretically proved that our method guarantees safety (forward invariance of the safe set) and performance (bounded tracking error and smoother trajectories). It has been numerically validated that the proposed method is computationally efficient for non-control-affine systems.
An Alternate Policy Gradient Estimator for Softmax Policies

Policy gradient (PG) estimators are ineffective in dealing with softmax policies that are sub-optimally saturated, which refers to the situation when the policy concentrates its probability mass on sub-optimal actions. Sub-optimal policy saturation may arise from bad policy initialization or sudden changes in the environment that occur after the policy has already converged. Current softmax PG estimators require a large number of updates to overcome policy saturation, which causes low sample efficiency and poor adaptability to new situations. To mitigate this problem, we propose a novel PG estimator for softmax policies that utilizes the bias in the critic estimate and the noise present in the reward signal to escape the saturated regions of the policy parameter space. Our theoretical analysis and experiments, conducted on bandits and various reinforcement learning environments, show that this new estimator is significantly more robust to policy saturation.
Research on the Construction and Service Mode Innovation of Open Mobile Knowledge Base in Higher Vocational Colleges

This article focuses on the construction of open mobile knowledge base (OMKB) in higher vocational colleges (HVCs), and adopts the methodologies of documental analysis, questionnaires, surveys and case studies to analyze OMKB in HVCs in China. Then, through the analysis of domestic and foreign data, it can be seen that the current storage volume of knowledge base of scientific research results in Chinese universities is far below the level of western developed countries. Conducting the construction of OMKB in HVCs can better comply with the trend of social development, meet the knowledge needs of the public, and form the complementarity between different institutions. However, in reality, there are still some problems and deficiencies in the OMKB service mode of HVCs, which emphasize information collection rather than knowledge application, paper literature rather than electronic data, on-campus service rather than social service, and existing knowledge rather than creative consciousness. Therefore, on the basis of full investigation and analysis, this paper proposes that the optimization path of the OMKB service mode in China›s HVCs is: exploring the joint mode of cooperation between universities and society, and operation and management, improving the system platform and funding sources, and making good storage and sharing of resources. The innovation of the OMKB service mode of HVCs in China can achieve the following: carrying out innovation and improvement, laying social foundation, promoting innovation of resource evaluation system, protecting the innovation of intellectual property mode, co-construction of the HVCs’ alliance knowledge base system innovation, technological innovation, - applying the advanced technology of CRIS, helping China’s HVCs’ knowledge base construction transformation and upgrading.
A Deep Learning Approach to Integrate Human-Level Understanding in a Chatbot

In recent times, a large number of people have been involved in establishing their own businesses. Unlike humans, chatbots can serve multiple customers at a time, are available 24/7 and reply in less than a fraction of a second. Though chatbots perform well in task-oriented activities, in most cases they fail to understand personalized opinions, statements or even queries which later impact the organization for poor service management. Lack of understanding capabilities in bots disinterest humans to continue conversations with them. Usually, chatbots give absurd responses when they are unable to interpret a user's text accurately. Extracting the client reviews from conversations by using chatbots, organizations can reduce the major gap of understanding between the users and the chatbot and improve their quality of products and services.Thus, in our research we incorporated all the key elements that are necessary for a chatbot to analyse and understand an input text precisely and accurately. We performed sentiment analysis, emotion detection, intent classification and named-entity recognition using deep learning to develop chatbots with humanistic understanding and intelligence. The efficiency of our approach can be demonstrated accordingly by the detailed analysis.
Skyrmion based energy efficient straintronic physical reservoir computing

Physical Reservoir Computing (PRC) is an unconventional computing paradigm, which exploits nonlinear dynamics of reservoir blocks to perform recognition and classification tasks. Here we show with simulations that patterned thin films hosting several skyrmions, particularly one, two, four and nine skyrmions, can implement energy efficient reservoir computing. This reservoir computing (RC) block is based on nonlinear breathing dynamics of skyrmions, which are coupled to each other through dipole interaction and spin waves, in response to a voltage generated strain. This nonlinear and coupled magnetization dynamics is exploited to perform temporal pattern recognition. Two performance metrics, namely Short-Term Memory (STM) and Parity Check (PC) capacity are studied to demonstrate the potential of such skyrmion based PRC in addition to showing it can classify sine and square waves with 100% accuracy. Furthermore, our study demonstrates that nonlinear magnetization dynamics and interaction through spin waves and dipole coupling have a strong influence on STM and PC capacity, thus explaining the role of physical interactions in a dynamical system on its ability to perform Reservoir Computing (RC).
Contour-guided Image Completion with Perceptual Grouping

Humans are excellent at perceiving illusory outlines. We are readily able to complete contours, shapes, scenes, and even unseen objects when provided with images that contain broken fragments of a connected appearance. In vision science, this ability is largely explained by perceptual grouping: a foundational set of processes in human vision that describes how separated elements can be grouped. In this paper, we revisit an algorithm called Stochastic Completion Fields (SCFs) that mechanizes a set of such processes -- good continuity, closure, and proximity -- through contour completion. This paper implements a modernized model of the SCF algorithm, and uses it in an image editing framework where we propose novel methods to complete fragmented contours. We show how the SCF algorithm plausibly mimics results in human perception. We use the SCF completed contours as guides for inpainting, and show that our guides improve the performance of state-of-the-art models. Additionally, we show that the SCF aids in finding edges in high-noise environments. Overall, our described algorithms resemble an important mechanism in the human visual system, and offer a novel framework that modern computer vision models can benefit from.
Uncalibrated Models Can Improve Human-AI Collaboration

In many practical applications of AI, an AI model is used as a decision aid for human users. The AI provides advice that a human (sometimes) incorporates into their decision-making process. The AI advice is often presented with some measure of "confidence" that the human can use to calibrate how much they depend on or trust the advice. In this paper, we present an initial exploration that suggests showing AI models as more confident than they actually are, even when the original AI is well-calibrated, can improve human-AI performance (measured as the accuracy and confidence of the human's final prediction after seeing the AI advice). We first train a model to predict human incorporation of AI advice using data from thousands of human-AI interactions. This enables us to explicitly estimate how to transform the AI's prediction confidence, making the AI uncalibrated, in order to improve the final human prediction. We empirically validate our results across four different tasks--dealing with images, text and tabular data--involving hundreds of human participants. We further support our findings with simulation analysis. Our findings suggest the importance of jointly optimizing the human-AI system as opposed to the standard paradigm of optimizing the AI model alone.
Germ FBI Transforms

This chapter is devoted to a special class of the integral transforms introduced in Ch. 21, generalizing the FBI transformand designed for the specific purpose of analyzing the wave-front set of a distribution.
Omnisolver: an extensible interface to Ising spin glass solvers

We introduce a new framework for implementing Binary Quadratic Model (BQM) solvers called omnisolver. The framework provides an out-of-the-box dynamically built command-line interface as well as input/output system, thus heavily reducing the effort required for implementing new algorithms for solving BQMs. The proposed software should be of benefit for researchers focusing on quantum annealers or discrete optimization algorithms as well as groups utilizing discrete optimization as a part of their daily work. We demonstrate the ease of use of the proposed software by presenting a step-by-step, concise implementation of an example plugin.
Toward Causal-Aware RL: State-Wise Action-Refined Temporal Difference

Although it is well known that exploration plays a key role in Reinforcement Learning (RL), prevailing exploration strategies for continuous control tasks in RL are mainly based on naive isotropic Gaussian noise regardless of the causality relationship between action space and the task and consider all dimensions of actions equally important. In this work, we propose to conduct interventions on the primal action space to discover the causal relationship between the action space and the task reward. We propose the method of State-Wise Action Refined (SWAR), which addresses the issue of action space redundancy and promote causality discovery in RL. We formulate causality discovery in RL tasks as a state-dependent action space selection problem and propose two practical algorithms as solutions. The first approach, TD-SWAR, detects task-related actions during temporal difference learning, while the second approach, Dyn-SWAR, reveals important actions through dynamic model prediction. Empirically, both methods provide approaches to understand the decisions made by RL agents and improve learning efficiency in action-redundant tasks.