Miracle-dz/UniFAR-FTUs huggingface dataset:
Generating Stylistic and Personalized Dialogues for Virtual Agents in Narratives

Virtual agents interact with each other through dialogues in various types of narratives (e.g. films). In this paper, we propose an approach on the basis of DialoGPT pre-trained language model, which explores the impact of dialogue generation with different levels of agents’ personalities derived from narrative films based on the Big-Five model, as well as with three different embedding methods. From the experimental results using automatic metrics and human user evaluation, we investigate and analyze the impact of different settings on narrative dialogue generation. We demonstrate that our approach is able to generate dialogues with increased variety that correctly reflect the corresponding target personality.
Learning Properties in Simulation-Based Games

Empirical game-theoretic analysis (EGTA) is primarily concerned with learning equilibria of simulation-based games. Recent statistical approaches have tackled this problem by first learning a uniform approximation of the game’s utilities, and then applying precision-recall theorems: i.e., all equilibria of the true game are approximate equilibria in the estimated game, and vice-versa. In this work, we generalize this approach to all game properties that are well-behaved (i.e., Lipschitz continuous in utilities), including regret (which defines Nash and correlated equilibria), adversarial values, power-mean welfare, and Gini social welfare. We show that, given a well-behaved welfare function, while optimal welfare is well-behaved, the welfare of optimal (i.e., welfare-maximizing or minimizing) equilibria is not well behaved. We thus define a re-lated property based on a Lagrangian relaxation of the equilibrium constraints that is well behaved. We call this property Λ -stable welfare. As determining the welfare of an optimal equilibrium is an essential step in computing the price of anarchy, we conclude with a discussion of an alternative, more stable notion of anarchy based on Λ -stable welfare, which we call the anarchy gap.
On Autonomy and Meaning in Human-Automation Interaction

Progress in automation will change the way we use and experience technology. The complex, unpredictable, and proactive nature of these technologies will transform them from tools controlled by humans to counterparts in dialogue with them. The existing models and frameworks on human-automation interaction (HAI), however, are insufficient to define this transition. They are mostly performance-oriented and neglect the experiential and meaning-oriented aspects of HAI, or highly rely on principles of direct manipulation limiting the affordances of automation. This creates a demand for alternative interaction paradigms based on quasi-social interaction. This paper elaborates on the problems of the existing frameworks and hints at new forms of interaction that consider experiential and performance goals.
Hierarchical Reinforcement Learning for Ad Hoc Teaming

When performing collaborative tasks with new unknown team-mates, humans are particularly adept at adapting to their collabo-rator and converging toward an aligned strategy. However, state of the art autonomous agents still do not have this capability. We propose that a critical reason for this disconnect is that there is an inherent hierarchical structure to human behavior that current agents lack. In this paper, we explore the use of hierarchical reinforcement learning to train an agent that can navigate the complexities of ad hoc teaming at the same level of abstraction as humans. Our results demonstrate that when paired with humans, our Hierarchical Ad Hoc Agent (HAHA) outperforms all baselines on both the team’s objective performance and the human’s perception of the agent.
Formally-Sharp DAgger for MCTS: Lower-Latency Monte Carlo Tree Search using Data Aggregation with Formal Methods

We study how to efficiently combine formal methods, Monte Carlo Tree Search (MCTS), and deep learning in order to produce high-quality receding horizon policies in large Markov Decision processes (MDPs). In particular, we use model-checking techniques to guide the MCTS algorithm in order to generate offline samples of high-quality decisions on a representative set of states of the MDP. Those samples can then be used to train a neural network that imitates the policy used to generate them. This neural network can either be used as a guide on a lower-latency MCTS online search, or alternatively be used as a full-fledged policy when minimal latency is required. We use statistical model checking to detect when additional samples are needed and to focus those additional samples on configurations where the learnt neural network policy differs from the (computationally-expensive) offline policy. We illustrate the use of our method on MDPs that model the Frozen Lake and Pac-Man environments -- two popular benchmarks to evaluate reinforcement-learning algorithms.
Semi-Popular Matchings and Copeland Winners

Given a graph $G = (V,E)$ where every vertex has a weak ranking over its neighbors, we consider the problem of computing an optimal matching as per agent preferences. Classical notions of optimality such as stability and its relaxation popularity could fail to exist when $G$ is non-bipartite. In light of the non-existence of a popular matching, we consider its relaxations that satisfy universal existence. We find a positive answer in the form of semi-popularity. A matching $M$ is semi-popular if for a majority of the matchings $N$ in $G$, $M$ does not lose a head-to-head election against $N$. We show that a semi-popular matching always exists in any graph $G$ and complement this existence result with a fully polynomial-time randomized approximation scheme (FPRAS). A special subclass of semi-popular matchings is the set of Copeland winners -- the notion of Copeland winner is classical in social choice theory and a Copeland winner always exists in any voting instance. We study the complexity of computing a matching that is a Copeland winner and show there is no polynomial-time algorithm for this problem unless $\mathsf{P} = \mathsf{NP}$.
Model-Based Reinforcement Learning for Auto-bidding in Display Advertising

Real-time bidding (RTB) achieves outstanding success in online display advertising, which has become one of the most influential businesses. Given historical ad impressions under the second price auction mechanism, the advertiser’s optimal bidding strategy is determined by the core parameter corresponding to the optimal solution of a constrained optimization problem. However, the sequentially arrived impressions in online display advertising make it highly non-trivial to obtain the optimal core parameter in advance without knowing the complete impression set. For this reason, recent methods have generally transformed the core parameter determination problem into a sequential parameter adjustment problem and solved it using reinforcement learning (RL). This paper proposes a simple and effective M odel-B ased A utomatic B idding algorithm, MBAB , which explicitly models the uncertainty of the dynamic auction environment and then uses the dynamic programming algorithm to obtain the current optimal adjustment of the core parameter. MBAB can avoid burdensome simulated environment construction and is more suitable for production deployment without the thorny sim-to-real issue than model-free methods. Furthermore, MBAB uses the optimal bidding formula to carry out coarse-grained modeling of the online market environment to alleviate the scalability problem caused by fine-grained environment modeling of previous model-based methods. In order to accurately describe the impression distribution and non-stationarity of the on-line market environment, we introduce the probabilistic modeling method and propose a novel monotonicity constraint to regulate the model output. Numerical experiments show that the proposed MBAB substantially outperforms existing baselines on various constrained RTB tasks in the production environment.
Learning Constraints From Human Stop-Feedback in Reinforcement Learning

We investigate an approach for enabling a reinforcement learning agent to learn about dangerous states or constraints from stop-feedback preventing the agent from taking any further, potentially dangerous, actions. Such feedback could be provided by human supervisors overseeing the RL agent’s behavior while carrying out some complex tasks. To enable the RL agent to learn from the supervisor’s feedback, we propose a probabilistic model for approximating how the supervisor’s feedback could have been generated and consider a Bayesian approach for inferring dangerous states. We evaluated our approach using an OpenAI Safety Gym environment and demonstrated that our agent can effectively infer the imposed safety constraints. Furthermore, we conducted a user study to validate our human-inspired feedback model and to obtain insights into the human provision of stop-feedback
Markov Aggregation for Speeding Up Agent-Based Movement Simulations

In this work, we investigate Markov aggregation for agent-based models (ABMs). Specifically, if the ABM models agent movements on a graph, if its ruleset satisfies certain assumptions, and if the aim is to simulate aggregate statistics such as vertex populations, then the ABM can be replaced by a Markov chain on a comparably small state space. This equivalence between a function of the ABM and a smaller Markov chain allows to reduce the computational complexity of the agent-based simulation from being linear in the number of agents, to being constant in the number of agents and polynomial in the number of locations. We instantiate our theory for a recent ABM for forced migration ( Flee ). We show that, even though the rulesets of Flee violate some of our necessary assumptions, the aggregated Markov chain-based model, MarkovFlee , achieves comparable accuracy at substantially reduced computational cost. Thus, MarkovFlee can help NGOs and policy makers forecast forced migration in certain conflict scenarios in a cost-effective manner, contributing to fast and efficient delivery of humanitarian relief.
Learnability with PAC Semantics for Multi-agent Beliefs

The tension between deduction and induction is perhaps the most fundamental issue in areas such as philosophy, cognition, and artificial intelligence. In an influential paper, Valiant recognized that the challenge of learning should be integrated with deduction. In particular, he proposed a semantics to capture the quality possessed by the output of probably approximately correct (PAC) learning algorithms when formulated in a logic. Although weaker than classical entailment, it allows for a powerful model-theoretic framework for answering queries. In this paper, we provide a new technical foundation to demonstrate PAC learning with multi-agent epistemic logics. To circumvent the negative results in the literature on the difficulty of robust learning with the PAC semantics, we consider so-called implicit learning where we are able to incorporate observations to the background theory in service of deciding the entailment of an epistemic query. We prove correctness of the learning procedure and discuss results on the sample complexity, that is how many observations we will need to provably assert that the query is entailed given a user-specified error bound. Finally, we investigate under what circumstances this algorithm can be made efficient. On the last point, given that reasoning in epistemic logics especially in multi-agent epistemic logics is PSPACE-complete, it might seem like there is no hope for this problem. We leverage some recent results on the so-called Representation Theorem explored for single-agent and multi-agent epistemic logics with the only knowing operator to reduce modal reasoning to propositional reasoning.
From Scripts to RL Environments: Towards Imparting Commonsense Knowledge to RL Agents

Text-based games provide a framework for developing natural language understanding and commonsense knowledge about the world in Reinforcement Learning (RL) based agents. Existing text-based environments often rely on fictional situations and characters to create a gaming framework and are far from real-world scenarios. In this paper, we introduce ScriptWorld : A text-based environment for teaching agents about real-world daily chores and hence imparting commonsense knowledge. To the best of our knowledge, it is the first interactive text-based gaming framework that consists of daily real-world human activities created using scripts dataset. We release the gaming environment and perform a detailed analysis of the proposed environment: https://github.com/Exploration-Lab/ScriptWorld.
SocialLight: Distributed Cooperation Learning towards Network-Wide Traffic Signal Control

Many recent works have turned to multi-agent reinforcement learning (MARL) for adaptive traffic signal control to optimize the travel time of vehicles over large urban networks. However, achieving effective and scalable cooperation among junctions (agents) remains an open challenge, as existing methods often rely on extensive, non-generalizable reward shaping or on non-scalable centralized learning. To address these problems, we propose a new MARL method for traffic signal control, SocialLight, which learns cooperative traffic control policies by distributedly estimating the individual marginal contribution of agents on their local neighborhood. SocialLight relies on the Asynchronous Actor Critic (A3C) framework, and makes learning scalable by learning a locally-centralized critic conditioned over the states and actions of neighboring agents, used by agents to estimate individual contributions by counterfactual reasoning. We further introduce important modifications to the advantage calculation that help stabilize policy updates. These modifications decouple the impact of the neighbors' actions on the computed advantages, thereby reducing the variance in the gradient updates. We benchmark our trained network against state-of-the-art traffic signal control methods on standard benchmarks in two traffic simulators, SUMO and CityFlow. Our results show that SocialLight exhibits improved scalability to larger road networks and better performance across usual traffic metrics.
A Scalable Opponent Model Using Bayesian Learning for Automated Bilateral Multi-Issue Negotiation

Learning an opponent’s preference is critical to achieving a win-win situation in automated bilateral multi-issue negotiations. Most of the existing opponent preference-learning techniques are not scalable to many kinds of opponents with different strategies due to their strong assumptions on an opponent’s concession pattern. This study enables a more general assumption into the Bayesian-learning-based opponent model to address the mentioned disadvantage. The proposed method is experimentally compared with state-of-the-art opponent models and found to have higher accuracy and greater scalability in most cases.
Time Series Models to Simulate and Forecast Wind Speed and Wind Power

Abstract A general approach for modeling wind speed and wind power is described. Because wind power is a function of wind speed, the methodology is based on the development of a model of wind speed. Values of wind power are estimated by applying the appropriate transformations to values of wind speed. The wind speed modeling approach takes into account several basic features of wind speed data, including autocorrelation, non-Gaussian distribution, and diurnal nonstationarity. The positive correlation between consecutive wind speed observations is taken into account by fitting an autoregressive process to wind speed data transformed to make their distribution approximately Gaussian and standardized to remove diurnal nonstationarity. As an example, the modeling approach is applied to a small set of hourly wind speed data from the Pacific Northwest. Use of the methodology for simulating and forecasting wind speed and wind power is discussed and an illustration of each of these types of applications is presen.
Cognitive Bias-Aware Dissemination Strategies for Opinion Dynamics with External Information Sources

The opinions of members of a population are influenced by opinions of their peers, their own internal predispositions, and information from external sources such as the media. Agents might perceive the received information differently due to various cognitive biases. In this paper, we propose a model of opinion evolution that uses prospect theory to represent the perception of information provided by an external source. Using the proposed model, we study the problem of selecting dissemination strategies for the external source to adopt in order to drive the opinions of individuals toward a desired value. As the initial predispositions of agents and functions characterizing agents’ perceptions of information disseminated might be unknown to the source, we estimate the unknown terms in the dynamics and find the optimal strategy by leveraging Gaussian process learning. Our simulations on three different widely-used large graph networks demonstrate that the external source can effectively drive a larger fraction of opinions towards a desired value by using a prospect-theory-based dissemination strategies.
The Grapevine Web: Analysing the Spread of False Information in Social Networks with Corrupted Sources

We study the problem of noisy information propagation in networks, where a small number of sources send messages across the network and agents use Bayesian updates to make inferences about the state of the world from the received messages. We provide upper bounds on the total number of sources necessary for learning on a given network and refine the bound in the case of small-world networks. We then extend the model to include an adversarial attacker, who can corrupt some of the information sources. We find that there is an optimal greedy attacking strategy in the case of a single learner, while the multi-learner case is not always solved optimally using greedy approaches. However, despite the influence function not being submodular, we show that the greedy algorithm performs well in practice. We also show that much simpler heuristics, which only look at centrality measures, can also provide a good basis to calculate successful attacking strategies. Finally we analyse the loss of optimality in the case when the attacker has incomplete information about the network and has to estimate the influence of source corruption heuristically. We use real-world social networks, as well as random network models, to empirically evaluate the effectiveness of attacking strategies and suggest a variety of measures to counteract them.
The Source of Desire: Personal Identityasa Drive for Agent Behavior

The central component in the design of our artificial agent behavior generation system, la VIDA , is its drive mechanism. This drive, which we have named an identity profile , is a collection of roles, values, beliefs, and attitudes analogous to and inspired by the human identity. This identity profile acts as an intrinsic source of motivation to drive long-term, autonomous agent behavior. In this paper we explore the representation and function of an early design for the identity profile. We also discuss how the identity profile interacts with a personality inventory and commonsense knowledge graph to assign traits and associated strengths to the agent. These values can then be mapped to three behavior intensities to augment agent actions.
Hiking up that HILL with Cogment-Verse: Train & Operate Multi-agent Systems Learning from Humans

As more AI systems are deployed, humans are increasingly required to interact with them in multiple settings. However, such AI systems seldom learn from these interactions with humans, which provides an important opportunity to improve from human expertise and context awareness. Several recent results in the fields of reinforcement learning (RL) and human-in-the-loop learning (HILL) show that AI agents can perform better when humans are involved in their training process. Humans can provide rewards to the agent, demonstrate tasks, design curricula, or act directly in the environment, but these potential performance improvements also come with architectural, functional design, and engineering complexities. This paper discusses Cogment, a unifying open-source framework that introduces a formalism to support a variety of human(s)-agent(s) collaboration topologies and training approaches. Cogment addresses the complexity of training with humans within a production-ready platform. On top of Cogment, we introduce Cogment Verse a research platform dedicated to the research community to facilitate the implementation of HILL and Multi-Agent RL experiments. With these platforms, our end goal is to enable the generalization of intelligence ecosystems where AI agents and humans learn from each other and collaborate to address increasingly complex or sensitive use cases. The video demonstration is available at https://youtu.be/v-K0DqIL9K0
Citizen-Centric Multiagent Systems

Advances in multiagent systems (MAS) have the potential to solve critical societal challenges. For example, MAS techniques for efficient resource allocation can help us implement cleaner and more efficient forms of on-demand mobility; social choice methods can support us in deciding how to trade off energy use and comfort in smart buildings; and task coordination methods can be used to respond to disasters in an effective and resilient manner. However, the benefits of these approaches can only be realised if citizen end users are able to trust these emerging multiagent systems. To achieve this, a citizen-centric approach needs to be taken. This places citizens at the heart of the design, development and deployment of trustworthy multiagent systems. We present open research challenges in this area, put forward key application domains for citizen-centric MAS (C-MAS) and discuss collaborative research opportunities.
Enhancing Reinforcement Learning Agents with Local Guides

This paper addresses the problem of integrating local guide policies into a Reinforcement Learning agent. For this, we show how to adapt existing algorithms to this setting before introducing a novel algorithm based on a noisy policy-switching procedure. This approach builds on a proper Approximate Policy Evaluation (APE) scheme to provide a perturbation that carefully leads the local guides towards better actions. We evaluated our method on a set of classical Reinforcement Learning problems, including safety-critical systems where the agent cannot enter some areas at the risk of triggering catastrophic consequences. In all the proposed environments, our agent proved to be efficient at leveraging those policies to improve the performance of any APE-based Reinforcement Learning algorithm, especially in its first learning stages.