Reviewerly/exHarmony huggingface dataset:
Online Distributed Learning with Quantized Finite-Time Coordination

In this paper we consider online distributed learning problems. Online distributed learning refers to the process of training learning models on distributed data sources. In our setting a set of agents need to cooperatively train a learning model from streaming data. Differently from federated learning, the proposed approach does not rely on a central server but only on peer-to-peer communications among the agents. This approach is often used in scenarios where data cannot be moved to a centralized location due to privacy, security, or cost reasons. In order to overcome the absence of a central server, we propose a distributed algorithm that relies on a quantized, finite-time coordination protocol to aggregate the locally trained models. Furthermore, our algorithm allows for the use of stochastic gradients during local training. Stochastic gradients are computed using a randomly sampled subset of the local training data, which makes the proposed algorithm more efficient and scalable than traditional gradient descent. In our paper, we analyze the performance of the proposed algorithm in terms of the mean distance from the online solution. Finally, we present numerical results for a logistic regression task.
Physics-Model-Regulated Deep Reinforcement Learning Towards Safety &amp; Stability Guarantees

Deep reinforcement learning (DRL) has demonstrated impressive success in solving complex control tasks by synthesizing control policies from data. However, the safety and stability of applying DRL to safety-critical systems remain a primary concern and challenging problem. To address the problem, we propose the Phy-DRL: a novel physics-model-regulated deep reinforcement learning framework. The Phy-DRL is novel in two architectural designs: a physics-model-regulated reward and residual control, which integrates physics-model-based control and data-driven control. The concurrent designs enable the Phy-DRL the mathematically provable safety and stability guarantees. Finally, the effectiveness of the Phy-DRL is validated by an inverted pendulum system. Additionally, the experimental results demonstrate that the Phy-DRL features remarkably accelerated training and enlarged reward.
Min-Time Coverage in Constricted Environments with Arbitrary Guidepath Networks

In a recent research program, we have undertaken the investigation of robotic traffic management problems arising when a fleet of networked mobile robots is employed in the support of certain coverage tasks that take place in physically constricted environments. But our past investigation of these problems is restricted to the case where the guidepath networks supporting the robot traffic have a dendritic topology. In the current work, we extend the investigation of the considered problems to the case where the underlying guidepath networks have an arbitrary topology. We provide (i) detailed descriptions of the considered problems in this new operational setting, (ii) analytical characterizations of these problems that take the form of integer programming formulations, and (iii) strong combinatorial relaxations for the derived formulations that are applicable to larger problem instances. A numerical experiment presented in the last part of the manuscript demonstrates and assesses the efficacy and the tractability of the analytical developments. We also notice that the undertaken extension of the past results is a nontrivial task, for the reasons that are explained in the manuscript. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> This work was partially supported by NSF grant ECCS-1707695.
Estimation of Dynamic Gaussian Processes

Gaussian processes provide a compact representation for modeling and estimating an unknown function, that can be updated as new measurements of the function are obtained. This paper extends this powerful framework to the case where the unknown function dynamically changes over time. Specifically, we assume that the function evolves according to an integro-difference equation and that the measurements are obtained locally in a spatial sense. In this setting, we will provide the expressions for the conditional mean and covariance of the process given the measurements, which results in a generalized estimation framework, for which we coined the term Dynamic Gaussian Process (DGP) estimation. This new framework generalizes both Gaussian process regression and Kalman filtering. For a broad class of kernels, described by a set of basis functions, fast implementations are provided. We illustrate the results on a numerical example, demonstrating that the method can accurately estimate an evolving continuous function, even in the presence of noisy measurements and disturbances.
Scalable Robust Multi-Agent Reinforcement Learning for Model Uncertainty

A robust multi-agent reinforcement learning (MARL) algorithm using a nature actor has been shown to be effective in finding a robust Nash equilibrium (NE) of a Markov game with model uncertainty. However, since a game-size scaling increases the search space and challenges reaching the NE, the robust property of the algorithm is reduced in environments with many agents. This paper proposes an evolutionary diversity-maintaining population curriculum (EDPC) framework with a robust attention-based multi-agent deep deterministic policy gradient (RA-MADDPG) algorithm, which enables an efficient robust NE search by a structured search space expansion. In the EDPC framework, the MARL divides into several stages, and when moving on to the next stage, a population consisting of larger games is made with two parent games from the previous stage. We introduce reward-proportionate parent selection and reward-guided mutation methods to continue reinforcing superior agents and maintain the diversity of the population. Furthermore, the RA-MADDPG is used to solve the robust Markov game at each stage with nature actors with attention-based architectures. The scalability and robustness of the proposed method are evaluated for different numbers of agents and levels of model uncertainty.
Data-Driven Reachability Analysis for Gaussian Process State Space Models

Gaussian process state space models are becoming common tools for the analysis and design of nonlinear systems with uncertain dynamics. When designing control policies for these systems, safety is an important property to consider. In this paper, we provide safety guarantees by computing finite-horizon forward reachable sets for Gaussian process state space models. We use data-driven reachability analysis to provide exact probability measures for state trajectories of arbitrary length, even when no data samples are available. We investigate two numerical examples to demonstrate the power of this approach, such as providing highly non-convex reachable sets and detecting holes in the reachable set.
Estimation of Quantum Channels Using Neural Networks

Quantum process tomography is an essential task for characterizing the dynamics of quantum systems and achieving precise quantum control. In this work, we propose a machine learning-based quantum process tomography method to reconstruct the Choi matrices of quantum channels from the measurements of the output states. Numerical results demonstrate that the proposed method exhibits a significant potential to achieve accurate reconstruction of different quantum channels.
Dynamics and Perturbations of Overparameterized Linear Neural Networks

Recent research in neural networks and machine learning suggests that using many more parameters than strictly required by the initial complexity of a regression problem can result in more accurate or faster-converging models - contrary to classical statistical belief. This phenomenon, sometimes referred to as “benign overfitting”, raises questions regarding in what other ways might overparameterization affect the properties of a learning problem. In this work, we investigate the effects of overfitting on the robustness of gradient-descent training when subject to uncertainty on the gradient estimation, which arises naturally if the gradient is estimated from noisy data or directly measured. Our object of study is a linear neural network with a single, arbitrarily wide, hidden layer and an arbitrary number of inputs and outputs, which can be equivalently written as an overparameterized matrix factorization problem. In this paper we solve the problem for the case where the input and output of our neural network are one-dimensional, deriving sufficient conditions for the robustness of our system based on necessary and sufficient conditions for convergence in the undisturbed case. We then show that the general overparameterized formulation introduces a set of spurious equilibria that lay outside the set where the loss function is minimized, and discuss directions of future work that might extend our current results for more general settings.
Transfer Learning for Barrier Certificates

A principled approach to safety verification of dynamical systems demands formal guarantees. Barrier certificates are an effective tool for searching safety proofs in the form of inductively verifiable invariants. However, finding barrier certificates is an expensive and time-consuming process that demands human expertise in selecting various templates, hyperparameters, and decision procedures. Is it possible to transfer the knowledge gained in finding a barrier certificate and control algorithm from a given environment (source environment) to a different but related environment (target environment)? This paper presents a transfer learning approach to adapt the barrier certificates (of any template) in the form of neural networks from the source to the target environment. We derive a validity condition to formally guarantee the correctness of network by leveraging its Lipschitz continuity. To demonstrate the effectiveness of our approach, we apply it to two case studies, namely the inverted pendulum, DC motor and Room temperature control. Our results show that transfer learning can successfully adapt barrier certificates from the source to the target environment, reducing the need for human expertise and speeding up the verification process.
Monotonic Model Improvement Self-Play Algorithm for Adversarial Games

The problem of solving strategy games has in-trigued the scientific community for centuries. In this paper, we consider two-player adversarial zero-sum symmetric games with zero information loss. Here, both players are continuously attempting to make decisions that will change the current game state to his/her advantage and hence the gains of one player are always equal to the losses of the other player. In this paper, we propose a model improvement self-play algorithm, where the agent iteratively switches roles to subdue the current adversary strategy. This monotonic improvement sequence leads to the ultimate development of a monolithic, competent absolute no-loss policy for the game environment. This tactic is the first of its kind in the setting of two-player adversarial games. Our approach could perform competitively and sometimes expertly in games such as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$4\mathrm{x}4$</tex> tic-tac-toe, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$5\mathrm{x}5$</tex> domineering, cram, and dots & boxes with a minimum number of moves.
Multi-UAV Trajectory Planning Problem Using the Difference of Convex Function Programming

Trajectory planning for a swarm of UAVs is known as a challenging nonconvex optimization problem, particularly due to a large number of collision avoidance constraints required for individual pairs of UAVs in the swarm. In this paper, we tackle this nonconvexity by leveraging the difference of convex function (DC) programming. We introduce the slack variables to relax and reformulate the collision avoidance conditions and employ the penalty function term to equivalently convert the problem into the DC form. Consequently, we construct a penalty DC algorithm in which we sequentially solve a set of convex optimization problems obtained by linearizing the collision avoidance constraint. The algorithm iteratively tightens the safety condition and reduces the objective cost of the planning problem and the additional penalty term. Numerical results demonstrate the effectiveness of the proposed approach in planning a large number of UAVs in congested space.
Automatic Decomposition of Reward Machines for Decentralized Multiagent Reinforcement Learning

In cooperative multiagent reinforcement learning (MARL), a team of agents learns to work together to complete a task. Centralized approaches to MARL quickly become intractable as the number of agents increases, necessitating decentralized learning algorithms which take advantage of task decompositions to train the agents individually. However, these task decompositions typically require careful human engineering. In this work, we develop algorithms to automatically decompose a team task into a collection of subtasks that can be used for decentralized reinforcement learning. We use reward machines-structured representations of reward functions-to encode team tasks and to automatically generate task decompositions that enforce the following three properties. 1) Task Consistency: We generate decompositions that are consistent with the team's task-if the agents individually learn to accomplish their subtasks, we guarantee that the composition of their learned behaviors will accomplish the original task. 2) Minimized Coordination: Inter-agent coordination during task execution can be costly. We minimize the coordination that's necessary to execute the decomposed tasks, which simplifies the decentralized learning problem by reducing each agent's interdependencies with its teammates. 3) Fairly Distributed: We maximize a weighted sum that balances the total utility of the agents and the fairness of the decomposition, which we define in terms of the distribution of assigned subtasks between the agents. Experimental results in three-agent and five-agent MARL tasks show the method's novel capabilities. The algorithm automatically generates task decompositions that are consistent with the team task, that reduce unnecessary coordination between the agents, and that take the agent's utility over subtasks into account. When used to define decentralized objectives, the generated task decompositions result in team policies that efficiently complete the task. Meanwhile, baseline decompositions yield policies that fail to complete the task.
Iteratively Preconditioned Gradient-Descent Approach for Moving Horizon Estimation Problems

Moving horizon estimation (MHE) is a state estimation method that has been extensively studied. The state estimates for the MHE problem are obtained by solving an approximation nonlinear optimization problem. This optimization process is known to be computationally challenging. This paper explores the idea of iteratively preconditioned gradient-descent (IPG) to solve the MHE issue to outperform the current solution methods in light of this limitation. To our knowledge, the preconditioning technique is employed for the first time in this research to speed up the critical MHE optimization stage and lower the computing cost. For a class of MHE problems, the proposed iterative approach's convergence guarantee is shown. Sufficient conditions for the MHE problem to be convex are also derived. Finally, the proposed method is implemented on a unicycle localization example. The simulation results demonstrate that the proposed approach can improve accuracy with reduced computational costs.
A Probabilistic Finite-State Automata Framework for Monitoring Long-Term Activities of Daily Living

Aging is an increasingly challenging healthcare issue with long-term sociopolitical implications, requiring more sophisticated management of healthcare services for the elderly. A key element in managing such services is the knowledge about a typical elderly person's Activities of Daily Living (ADLs). However, there is little study on the model-based objective and standardized ADLs assessment. This paper presents the concept of tracking the health status of the elderly by monitoring their individual patterns of ADLs: the specific pattern of the elderly model as probabilistic state-transition structures, and the transition probabilities in such a probabilistic model are considered as a representation of the health status of a typical older person. The typical pattern of ADLs changes with the health status due to changes in the residential environment, progressive aging, or the onset of certain diseases. Such changes are reflected (over time) in the model, and therefore monitoring the transition probabilities to assess ADLs represents a possible way to track the health status of an older person and to alert relevant health service professionals if some change that warrants subsequent careful medical attention.
Computationally Efficient Reinforcement Learning: Targeted Exploration leveraging Simple Rules

Model-free Reinforcement Learning (RL) generally suffers from poor sample complexity, mostly due to the need to exhaustively explore the state-action space to find well-performing policies. On the other hand, we postulate that expert knowledge of the system often allows us to design simple rules we expect good policies to follow at all times. In this work, we hence propose a simple yet effective modification of continuous actor-critic frameworks to incorporate such rules and avoid regions of the state-action space that are known to be suboptimal, thereby significantly accelerating the convergence of RL agents. Concretely, we saturate the actions chosen by the agent if they do not comply with our intuition and, critically, modify the gradient update step of the policy to ensure the learning process is not affected by the saturation step. On a room temperature control case study, it allows agents to converge to well-performing policies up to 6–7× faster than classical agents without computational overhead and while retaining good final performance.
A Coherent LQG Approach to Quantum Equalization

We propose a method to design a suboptimal, coherent quantum LQG controller to solve a quantum equalization problem. Our method involves reformulating the problem as a control problem and then designing a classical LQG controller and implementing it as a quantum system. Illustrative examples are included which demonstrate the algorithm for both active and passive systems, i.e., systems where the dynamics are described in terms of both position and momentum operators and systems with dynamics in terms of annihilation operators only.
Homomorphically Encrypted Gradient Descent Algorithms for Quadratic Programming

In this paper, we evaluate the different fully homomorphic encryption schemes, propose an implementation, and numerically analyze the applicability of gradient descent algorithms to solve quadratic programming in a homomorphic encryption setup. The limit on the multiplication depth of homomorphic encryption circuits is a major challenge for iterative procedures such as gradient descent algorithms. Our analysis not only quantifies these limitations on prototype examples, thus serving as a benchmark for future investigations, but also highlights additional trade-offs like the ones pertaining the choice of gradient descent or accelerated gradient descent methods, opening the road for the use of homomorphic encryption techniques in iterative procedures widely used in optimization based control. In addition, we argue that, among the available homomorphic encryption schemes, the one adopted in this work, namely CKKS, is the only suitable scheme for implementing gradient descent algorithms. The choice of the appropriate step size is crucial to the convergence of the procedure. The paper shows firsthand the feasibility of homomorphically encrypted gradient descent algorithms.
Distributionally Robust Behavioral Cloning for Robust Imitation Learning

Robust reinforcement learning (RL) aims to learn a policy that can withstand uncertainties in model parameters, which often arise in practical RL applications due to modeling errors in simulators, variations in real-world system dynamics, and adversarial disturbances. This paper introduces the robust imitation learning (IL) problem in a Markov decision process (MDP) framework where an agent learns to mimic an expert demonstrator that can withstand uncertainties in model parameters without additional online environment interactions. The agent is only provided with a dataset of state-action pairs from the expert on a single (nominal) dynamics, without any information about the true rewards from the environment. Behavioral cloning (BC), a supervised learning method, is a powerful algorithm to address the vanilla IL problem. We propose an algorithm for the robust IL problem that utilizes distributionally robust optimization (DRO) with BC. We call the algorithm DR-BC and show its robust performance against parameter uncertainties both in theory and in practice. We also demonstrate the empirical performance of our approach to addressing model perturbations on several MuJoCo continuous control tasks.
Optimal Transport Particle Filters

This paper is concerned with the theoretical and computational development of a new class of nonlinear filtering algorithms called optimal transport particle filters (OTPF). The algorithm is based on a recently introduced variational formulation of the Bayes' rule, which aims to find the Brenier optimal transport map between the prior and the posterior distributions as the solution to a stochastic optimization problem. On the theoretical side, the existing methods for the error analysis of particle filters and stability results for optimal transport map estimation are combined to obtain uniform error bounds for the filter's performance in terms of the optimization gap in solving the variational problem. The error analysis reveals a bias-variance trade-off that can ultimately be used to understand if/when the curse of dimensionality can be avoided in these filters. On the computational side, the proposed algorithm is evaluated on a nonlinear filtering example in comparison with the ensemble Kalman filter (EnKF) and the sequential importance resampling (SIR) particle filter.
Verification of Low-Dimensional Neural Network Control

We verify safety of a nonlinear continuous-time system controlled by a neural network controller. The system is decomposed into low-dimensional subsystems connected in a feedback loop. Our application is a rocket landing, and open-loop properties of the two-dimensional altitude subsystem are verified using worst-case simulations. Closed-loop safety properties (crash-avoidance) of the full system are obtained from composition of contracts for open-loop safety properties of subsystems in a fashion analogous to the small-gain theorem.