QUANTIFYING THE HUMAN-LIKENESS OF ARTIFICIALLY INTELLIGENT AGENTS USING STATISTICAL METHODS AND TECHNIQUES

An apparatus includes a processor configured to determine a first distribution associated with an artificial agent based on behavior associated with the artificial agent and a second distribution based on behavior of a user. The processor is further configured to generate a human-likeness similarity measurement by comparing the first distribution to the second distribution and modify the behavior of the artificial agent in response to the similarity measurement failing to satisfy a similarity threshold.

BACKGROUND

One mechanism for implementing artificial intelligence (AI) is through artificial agents, which are non-human entities that have knowledge of their environment and autonomously perform actions to achieve a goal(s). Artificial agents are becoming an integral part of many different technologies and industries. For example, vehicles implement artificial agents to perform autonomous driving operations, video games implement artificial agents as non-playable characters (NPC), and factories implement robots as artificial agents to autonomously perform various actions. In many instances, artificial agents are trained using various techniques to emulate human behavior in their environments. Therefore, evaluating the human-likeness of an agent's behavior often is an important part of the design and development of artificial agents.

DETAILED DESCRIPTION

Developing artificial agents capable of learning complex human-like behaviors is a goal of artificial intelligence research. Advancements in machine learning techniques, such as deep reinforcement learning, have enabled the development of highly skilled artificial agents capable of complex intelligent behavior. For example, in video games, artificial agents are increasingly deployed as non-playable characters (NPCs) that are designed to emulate human behavior and enhance the experience of human players. Although the design and development of highly skilled artificial agents has progressed, quantifying the human-likeness of artificial agents remains a challenge. For example, the believability of an artificial agent's behavior is often measured solely by the agent's proficiency at a given task, but proficiency alone is not sufficient to measure human-like behavior.

Various approaches have attempted to quantify/measure the believability of artificial agents' behavior but each of these approaches have various drawbacks. For example, one approach for evaluating artificial agent behavior involves human assessment (e.g., a Turing Test) of the agents. However, this approach is not practical for most environments given the speed, scalability, and cost limitations of human assessment. Another approach implements environment (domain) specific tests that rely on rules or heuristics to detect human-like behavior of agents. This approach requires a large amount of manual effort by domain experts, which is time consuming and costly. Also, the environment specific tests tend to coarsely evaluate differences in results rather than directly evaluating agent behavior and fail to capture fine-grained details of human-likeness (e.g., detecting cyclic behavior, counting collisions, or measuring the steps taken to achieve a goal, etc.). Other approaches implement machine learning to assess the behavior of artificial agents. However, the machine learning models are usually fit to a specific environment and are not easily generalizable.

The present disclosure describes embodiments of systems and methods for overcoming various issues, including computational cost, efficiency, and accuracy, associated with conventional techniques for quantifying the human-likeness of artificial agent behavior. As described in greater detail below, the similarity between the behavior of an artificial agent(s) and the behavior of a human counterpart(s) is evaluated and quantified using a non-parametric two-sample hypothesis testing technique. The non-parametric two-sample hypothesis testing technique uses a distribution of an artificial agent's state-action pairs (e.g., a trajectory) to determine if the behavior of the agent is human-like and further quantifies how human-like the behavior is. For example, the similarity of their respective behaviors is compared using a distribution of a human behavior sequence and a distribution of an artificial intelligence behavior sequence associated with the same or similar task/goal. The two-sample hypothesis testing technique outputs a similarity measurement quantifying the human-likeness of the artificial agent's behavior.

An advantage of the artificial agent behavior analysis techniques described herein is that they are domain/environment agnostic and are applicable to multiple different abilities associated with intelligent behavior unlike conventional analysis techniques, such as machine learning-based analysis techniques. Also, the analysis techniques described herein avoid the costs associated with human judges and do not require training/re-training or inference optimization required by techniques that implement deep neural networks (DNNs). Another advantage is that because the analysis techniques described herein are based on statistical distributions they can be easily parallelized and accelerated. Moreover, the results (e.g., similarity measurements) of the analysis techniques, in at least some embodiments, are used to improve or modify the artificial agents, improve the system or environment implementing artificial agents, a combination thereof, or the like. For example, the analysis results are used to rank or score artificial agents, train or re-train artificial agents for improving the human-likeness of the agents' behavior, or to adjust an artificial agent's behavior in real-time while executing in an operating environment (e.g., a video game, a vehicle, a robot, etc.). Other applications include optimizing the curriculum in a curriculum-based learning paradigm, evaluating the design of a reward signal, optimizing neural architectures, and early termination of training an agent when sufficient similarity is reached. It should be understood that the two-sample hypothesis techniques described herein are also applicable to evaluating the behavior of multiple artificial agents to determine the similarity of the behaviors between the multiple artificial agents. The results of this analysis is used, for example, to increase the diversity in behaviors in a multi-agent setting.

FIG.1illustrates an example operating environment100for evaluating the human-likeness of artificial agent behavior and modifying the artificial agents based on the evaluation. As shown inFIG.1, the operating environment100comprises one or more information processing systems102(shown as systems102-1to102-3) that are communicatively coupled by one or more networks104. It should be understood that the number of information processing systems102shown inFIG.1is for illustration purposes only. For example, a single system102or a lesser/greater number of information processing systems102are able to include the components and perform the operations described herein. Examples of information processing systems102include servers, desktop computers, laptop/notebook computers, mobile devices, or various other types of computing systems or devices. The network(s)104, in at least some embodiments, is implemented utilizing wired and or wireless networking mechanisms. For example, the network104, in at least some embodiments, comprises wireless communication networks, non-cellular networks such as Wi-Fi networks, public networks such as the Internet, private networks, and so on. The wireless communication networks support any wireless communication standard and include one or more networks based on such standards.

In the example shown inFIG.1, a first information processing system102-1comprises an artificial agent behavior analysis (AABA) system106that evaluates one or more behaviors of artificial agents108(shown as108-1and108-2) and, in at least some embodiments, modifies the behavior of an artificial agent108based on this evaluation. Artificial agent behavior is an autonomous action(s) taken by an artificial agent108during operation/execution in response to its environment. Examples of artificial agent behaviors include autonomous driving behaviors (e.g., lane negotiation, collision avoidance, turning, etc.), autonomous navigation of a virtual gaming environment, autonomous fabrication a product, autonomous interaction with a human, prediction, perception, reasoning, and so on.

The artificial agents108, in at least some embodiments, are implemented at the first information processing system102-1, one or more other information processing systems102, or a combination of both. For example, one or more agents108are implemented at a second information processing system102-2and a third information processing system102-3. The second information processing system102-2comprises an artificial agent training system110that uses training data112to train artificial agents108-1for autonomously performing one or more actions/behaviors in response to their environment. In one example, the artificial agent training system108is a machine learning (ML)-based training system. However, different techniques, such as Navigation Mesh and behavior trees, can be used to program or train an artificial agent108as well. The third information processing system102-3comprises one or more artificial agents108-2executing within an operating environment114, such as virtual environment (e.g., video game)114-1or a user-assistive environment114-2(e.g., virtual chat). In another example, the operating environment114is an entity, such as a vehicle114-3(manned or unmanned) or a robot114-4, implementing the artificial agents108-2. It should be understood the first and second information processing systems102-2,102-3are only provided as examples of the types of systems on which artificial agents108can be implemented. The AABA system106of the first information processing system102-1is able to analyze/evaluate different types of behaviors performed by different types of artificial agents across various domains and systems.

AlthoughFIG.1shows the AABA system106implemented at the first information processing system102-1, in other embodiments, the AABA system106is implemented as part of, or in conjunction with, the artificial agent training system110, the artificial agent operating environment114, another system/environment capable of implementing artificial agents108, a combination thereof, or the like. In at least some embodiments, the AABA system106, includes a behavior distribution module116, a behavior information transformation module118, a behavior similarity determination module120, and a dimensionality reduction module122. One or more of these modules, in at least some embodiments, are distributed across multiple information processing systems, and two or more of these modules can be combined. In at least some embodiments, the AABA system106includes one or more interfaces124, such as an artificial agent training system interface124-1and an artificial agent operating environment interface124-2. The AABA system106uses the interfaces124, for example, to interface and communicate with one or more systems comprising artificial behavior information126, such as artificial agent behavior information126-1and human behavior information126-2, or that apply the results128of the artificial agent behavior analysis performed by the AABA system106.

As described in greater detail below, the AABA system106uses one or more of the behavior distribution module116, the behavior information transformation module118, the behavior similarity determination module120, or the dimensionality reduction module122to perform an artificial agent behavior analysis that quantitively evaluates the similarity of the behaviors exhibited by artificial agents108to those exhibited by human counterparts (e.g., human users) in a domain-agnostic manner. The behavior analysis is a statistical method that uses a distribution of an agent's state-action pairs (e.g., a trajectory) to determine if the behavior of the agent is human-like and further quantifies how human-like the behavior is. For example, the AABA system106analyzes the human-like behavior (HLB of an artificial agent108based on comparing the distribution of the behavior between a distribution of a human behavior sequence and a distribution of an artificial intelligence behavior sequence associated with the same or similar task/goal. Based on the behavior analysis, the AABA system106generates behavior analysis results128that include, for example, a measure of similarity128-1(also referred to herein as a similarity measurement128-1) for each evaluated artificial agent108. The measure of similarity (human-likeness)128-1quantifies/measures how similar the behavior of the artificial agent108is to the behavior of a human(s) counterpart. For example, if the behavior being evaluated is navigating an environment of a video game to achieve a goal (e.g., reach an endpoint), the measure of similarity128-1quantifies the human-likeness of the artificial agent's movements in achieving this goal. In another example, the measure of similarity128-1quantifies the performance (in terms of human-like behavior) of a robot with respect to behaviors such as moving or performing a task. In a further example, the measure of similarity128-1quantifies how believable a human-assistive software agent or a non-playable character is when interacting with a human.

In at least some embodiments, the behavior analysis results128are applied by one or more systems to, for example, improve or enhance one or more artificial agents108, improve or enhance the system or environment implementing the artificial agent(s)108, a combination thereof, or the like. For example, the AABA system106or another system, such as the artificial agent training system110or the artificial agent operating environment114, uses the measure of similarity128-1to rank or score the artificial agents108that have been evaluated, to train or re-train artificial agents108for improving the human-likeness of the agents' behavior, or to adjust an artificial agent's behavior in real-time while executing in an operating environment114(e.g., a video game, a vehicle, a robot, etc.). Other applications include optimizing the curriculum in a curriculum-based learning paradigm, evaluating the design of a reward signal, optimizing neural architectures, and early termination of training an agent when sufficient similarity is reached.

For example,FIG.2illustrates in flow chart form, an overview of one example method200of applying the behavior analysis results128as part of an artificial agent training process. In this example, p is the similarity measurement128-1and γ is a predefined confidence level201. At block202, the training system110trains an artificial agent108(e.g., an AI/ML agent) using one or more training techniques, such as reinforcement learning (RL). During training, agent behavior information126-1associated with the artificial agent108is generated for one or more tasks (e.g., navigating an environment within a video game). At block204, the AABA system106analyzes the agent behavior information126-1with respect to human behavior information126-2associated with the same task. Human behavior information126-2, in at least some embodiments, includes behavior of an actual human counterpart or behavior of an artificial agent108controlled by a human counterpart. Based on this analysis, the AABA system106generates a similarity measurement (p)128-1quantifying the human-likeness of the artificial agent's behavior126-1. At block206, the AABA system106(or the training system110) determines if the similarity measurement p128-1satisfies a similarity threshold (γ)201(e.g., p≥γ). If so, at block210, the AABA system106instructs the training system110to stop training the AABA system106, which reduces training computational time and prevents overtraining of the artificial agent108. The process then exits at block210. The trained artificial agent(s)108is then implemented in one or more operating environments114. Otherwise, at block212the AABA system106instructs the training system110to adjust the behavior of the artificial agent108by continuing the training the artificial agent108, and the process returns to block202. In other embodiments, the training system110receives the similarity measurement p128-1and performs the operations at block206rather than the AABA system106.

In at least some embodiments, one or more of the techniques implemented by the AABA system106for evaluating artificial agent behavior are based on, for example, deep reinforcement learning and distribution analysis and comparison. Deep reinforcement learning is one technique for training/programming artificial agents108by a system, such as the training system110ofFIG.1. In one example, the standard formulation of reinforcement learning (RL) is a Markov decision process (MDP) in which an artificial agent108interacts with an environment ε according to a policy with the goal of maximizing its cumulative expected reward. Let πθ(a|s) denote the decision policy of a given artificial agent108, where πθmodels the conditional distribution over action a given state s and is parameterized by θ. In deep reinforcement learning (DRL), this conditional distribution is modeled using a deep neural network (DNN). The space of all actions and states can be denoted asand, respectively, such that a∈and s∈. At each time step t, the artificial agent108observes the current state st and samples an action ataccording to πθ. The environment ε then responds with a scalar reward rtthat reflects the value of that transition and a new state st+1, which is sampled from the transition dynamics of ε denoted by pε(st+1|s, a). Note that, given state s and action a, the transition dynamics satisfy the Markov property such that the probability distribution over the next state is conditionally independent of all previous states and actions. The objective under this paradigm is to learn the optimal parameters θ* that maximize the expected return, as given by EQ 1:

Here, pθ(τ) denotes the distribution over all possible trajectories τ when following policy πθ, and p(s0) denotes the initial state distribution. A sequence of state-action pairs spanning an initial state so to a terminal state sNis defined as an episode, and a continuous subsequence of an episode over a horizon of T≤N time steps is defined as a trajectory, which is denoted τ. In other words, each trajectory τ is defined by a sequence of state-action pairs over a horizon of T time steps such that τ=(s0, a0, s1, . . . , aT−1, sτ). Note that Σt=0Tγtrtrepresents the total return of trajectory τ over this time horizon where γ∈[0,1] is a discount factor used to ensure a finite return. It should be understood that other techniques can be used to train/program the artificial agents108.

One example of a distribution analysis and comparison technique implemented by the AABA system106is Maximum Mean Discrepancy (MMD). MMD is a class of kernel-based divergence metrics used to compute the distance between the projections of two high-dimensional data distributions. MMD distance is defined as:

The AABA system106empirically estimates this distance by independently drawing n samples from each distribution. Given that kernel k maps to a reproducing kernel Hilbert space, then the MMD distance given by EQ 2 is zero if and only if the distributions X and Y are identical. Here, x and y denote sample distributions of n elements independently drawn from X and Y, respectively, where x′ indicates that the sample has been shuffled andX,Y[k(x, y)] is estimated either pairwise or elementwise. EstimatingX,Y[k(x, y)] with pairwise distances such that

uses each sample point to maximum effect. However, the compute cost increases quadratically with the size of the sample distribution. As such, with larger volumes of data,X,Y[k(x, y)] can be estimated element-wise such that

which is intuitively less computationally expensive but more prone to sampling error as the result is dependent on the sampling order. Therefore, in at least some embodiments, the AABA system106uses the pairwise estimation ofX,Y[k(x, y)] with the standard Gaussian kernel and the Euclidean distance function such that k(x, y)=−exp(∥x−y∥2)/2σ2. Here, σ is referred to as the kernel bandwidth and is set to the median pairwise distance of the aggregated samples from X and Y. It should be understood that MMD is only one example of a distribution analysis and comparison technique implemented by the AABA system106and other techniques are applicable as well, such as Wasserstein Distances or Sinkhorn Divergences.

As described below, the AABA system106also implements one or more non-parametric statistical hypothesis testing techniques for evaluating artificial agent behavior. Statistical inference is the process of drawing conclusions about a population parameter or population distribution from sample distributions. Unlike parametric statistical inference, which estimates sample statistics to model a population distribution using assumptions about the data, non-parametric statistical inference analyzes the sample distribution directly. Among this class of tests and tools, sampling methods such as bootstrap resampling and permutation testing offer increased performance and generality over traditional methods without requiring distributional assumptions. With bootstrap resampling, the AABA system106repeatedly draws samples of size m with replacement from a sample distribution of size n to recompute a sample statistic. Although some bootstrap resampling methods set m=n, m out of n bootstrapping (i.e., m≤n) can yield more consistent results. Similar to bootstrap resampling, permutation tests do not require a priori knowledge of the data distribution. However, unlike bootstrap resampling, this class of tests resamples from the sample distribution without replacement. In at least some embodiments, the AABA system106implements one or more of bootstrap resampling techniques or permutation testing techniques for hypothesis testing, although other techniques are applicable as well.

FIG.3illustrates, in flow chart form, an example method300for evaluating artificial agent behavior. Although subprocesses of method300are illustrated and described in an example order, the method300is not limited to this particular order, and in some embodiments, certain processes may be performed in a different order, or concurrently rather than in sequential order, or may be omitted altogether. Moreover, although the method300is described in the example context of a video game environment and movement/navigation behaviors, it should be understood the techniques described herein are not limited to such an environment and behaviors.

As described below, the method300analyzes the similarity between artificial agent behavior and human counterpart behavior (or another artificial agent) using a non-parametric two-sample hypothesis testing technique, which is configured according to a behavior similarity hypothesis. In at least some embodiments, the behavior similarity hypothesis indicates that the behaviors of any two agents are sufficiently similar if the distributions over their respective behaviors, which are representable as episodes/trajectories, are sufficiently similar. Also, the method300is described in an example context where the artificial agents108implement a pre-trained decision policy πθthat drives the artificial agents108to complete a given task when deployed in an environment Σ, such as a video game environment. Similarly, the human counterpart(s) being compared to the artificial agents108in the method300independently completes the same task in the same environment Σ. For example, if the artificial agent108is an NPC in a video game environment, the human counterpart is independently given control over the same (or different) artificial agent(s)108to complete the same task in the same environment Σ. As such, in at least some embodiments, both the artificial agent108and the human counterpart are bound to the same initial state distribution p(s0) with the same transition dynamics pε(st+1|st, at). The distribution of the behaviors/trajectories induced by decision policy πθof the artificial agent108is defined as Pθ(τ) and the distribution of behaviors/trajectories induced by the human counterpart is defined as P*(τ). For brevity, these distributions are herein referred to as Pθand P*, respectively.

At block302, the AABA system106takes as input agent behavior information126-1for one or more artificial agents108and takes as input human behavior information126-2for one or more human counterparts. The behavior information126, in at least some embodiments, is stored locally at the information processing system comprising the AABA system106. In other embodiments, the behavior information126is stored remote from the information processing system comprising the AABA system106. The behavior information126, in at least some embodiments, is obtained from the artificial agent training system110, the operating environment114, human testers, human players, a combination thereof, or the like. For illustrative purposes, the behavior information126in this example represents navigation/movement behaviors of the artificial agent(s)108and human counterpart(s) within a three-dimensional (3D) space of a video game environment. Therefore, in this example, each movement identified by the behavior information126is defined by an x-coordinate, a y-coordinate, and a z-coordinate. It should be understood that other types of behavior are representable/definable by different characteristics or attributes.

Also, the agent behavior information126-1and the human behavior information126-2are each represented by one or more episodes301(illustrated as episode301-1and episode301-2). For example, if the behavior being evaluated is navigation within a virtual environment, such as a video game, the agent behavior information126-1and the human behavior information126-2each comprise one or more episodes301representing all the navigation movements of the artificial agent108and the human counterpart, respectively, for a given task. It should be understood that the behavior of an artificial agent108and a human counterpart is not limited to being represented as by an episode(s) and trajectories, as other techniques are applicable as well. Each episode301-1of the artificial agent behavior information126-1is comprised of trajectories τ303-1and each episode301-2of the human behavior information126-1is composed of trajectories τ303-2. As described above, a trajectory τ303is a continuous subsequence of an episode301over a horizon of T≤N time steps. In other words, each trajectory τ303is defined by a sequence of state-action pairs over a horizon of T time steps such that τ=(s0, a0, s1, . . . , aT−1, sτ). Therefore, a trajectory τ303is a slice/window of the episode comprising a subsequence of the behavior/actions performed the artificial agent108or human counterpart. For example, if an episode301represents 1000 movements taken by the artificial agent108, each trajectory τ303represents a subset of these 1000 movements.

At block304, the distribution module116of the AABA system106determines/generates distributions P303for the artificial agent behavior information126-1and the human behavior input126-2. For example, the distribution generation module116generates a distribution Pθ303-1for the artificial agent behavior information126-1and generates a distribution P*303-2for the human behavior information126-2. In this example, the distributions P305are over the space of trajectories303, e.g., Pθ(τ)301-1and P*(τ)301-2determined at block304.

At blocks306and308distribution module116and the behavior information transformation module118(referred to herein as transformation module118for brevity) of the AABA system106operate to generate a sample distribution X309-1for the artificial agent behavior information126-1and a sample distribution Y309-2for the human agent behavior information126-2. Sample distribution X309-1and sample distribution Y309-2are independently drawn from distribution Pθ305-1and distribution P*305-2, respectively. For example, at block306, let E denote a set of M episodes301independently collected from each of the artificial agent behavior information126-1and the human behavior information126-2. Let N denote the total number of time steps between the initial state so and the terminal state sNin a given episode. The distribution module116independently subsamples K trajectories303of length T from each episode301with replacement. In some instances, the length of an episode301can be heavily skewed. Therefore, to correct for any biases from larger episodes with more time steps, K is set to the length of the largest episode301in the given set. As such, the size of each sample distribution309(e.g., |X|, |Y|) generated at block308, in at least some embodiments, is M·K. This ensures that a trajectory drawn at random from the aggregated sample distribution has a uniform probability of being from any of the episodes301collected. Also, there is no point-to-point correspondence between sample distribution X309-1and sample distribution X309-2.

At block308, the transformation module118transforms the behavior information126-1of the artificial agent(s)108and the behavior information126-2of the human counterpart(s) from a first representation to a different representation, such as vectors307-1and images307-2). In at least some embodiments, this different representation is a high-dimensional representation of the behavior information126, such as a vector307-1, an image307-2, or the like. In one example, the transformation module118receives as input each sampled trajectory τ303generated by the distribution module116at block306. Then, for each sampled trajectory τ303, the transformation module118represents the sampled trajectory τ303as a vector307-1including the absolute three-dimensional (3D) location of the artificial agent108at each time step. The transformation module118uses the vector307-1to transform and visualize the sampled trajectory303into a two-dimensional (2D) rendering (image)307-2. The absolute 3D location of the artificial agent108can represent the minimal representation of the artificial agent's response to a given state. In this example, let ctbe the 3D Cartesian coordinates of an artificial agent108at time t and let c(τ) be the sequence of coordinates for a given sampled trajectory303such that c(τ)={c0, . . . , cT} and ct=(xt, yt, zt). The transformation module118projects a sampled trajectory τ303into a 2D rendering by using c(τ) to view the navigation of the artificial agent108from the z-axis and applying a min-max scaling over the (x, y) coordinates to project each vector of Cartesian coordinates into the range of [0, 1]. The transformed coordinates ({circumflex over (x)},ŷ) are defined as

respectively. Note that the minimum and maximum of each x and y coordinate space are determined over c(τ), which is the sequence of coordinates of the sampled trajectory τ303. The transformation module118scales these coordinates using the desired Height×Width (H×W) resolution as shown in EQ 3:

The resulting H×W image307-2is referred to as the top-down projection307-2of the sampled trajectory τ303, which is denoted as operator Iτ.

As objectives become increasingly complex, the episodes301can vary greatly and non-uniformly, thereby introducing artifacts. For example, the fine details of long, complex trajectories lose their salience in the projections, and the continuity of short smooth trajectories is disrupted, which introduces a sparse projection. Therefore, in at least some embodiments, the transformation module118introduces two modifications to the top-down trajectory projection process described above. In the first modification, the transformation module118exploits the conditional independence property of MDPs to subsample fixed-length trajectories303without replacement from each episode301using a finite time horizon denoted by T. In the second modification, the transformation module118linearly interpolates along c(τ) and dynamically shifts and scales the axes to increase the fidelity of the resulting image307-2.FIG.4shows various examples of projection output402generated by the transformation module118. In this example, the projection output402comprises one or more projections/images404(illustrated as projections404-1to404-4) representing navigation behavior of one or more artificial agents108and one or more human counterparts. In at least some embodiments, each sampled trajectory τ303determined for the behavior information126is represented as a projection/image307-2. In at least some embodiments, the output of the transformation module118is a sample distribution X309-1(of distribution Pθ305-1) comprising the collection of projections/images307-2generated for the artificial agent behavior information126-1and a sample distribution Y309-2(of distribution P*305-2) comprising the collection of projections/images307-2generated for the human behavior information126-2.

Referring to block308ofFIG.3, in another example, the transformation module118transforms the behavior information126into a different representation by vectorizing the sampled trajectories τ303representing the artificial agent behavior information126-1and the human behavior information126-2using the absolute 3D location of the artificial agent108at each time step, similar to the example described above. However, in this example, the transformation module118does not transform the vectors307-1representing the trajectories τ303into a 2D visual representation. Instead, the transformation module118generates the sample distribution X309-1and the sample distribution Y309-2directly from the vectors307-1representing the sampled trajectories τ302, which comprise x, y, and z coordinate data of the sampled trajectories τ303. For example, the transformation module118transforms each episode301-1and301-2representing the artificial agent behavior information126-1and the human behavior information126-2, respectively, into a distribution of movements by subsampling fixed-length trajectories τ303uniformly with replacement from each episode301using a finite time horizon denoted by T. More formally, let ctbe the 3D Cartesian coordinates of an artificial agent108at time t and let c(τ) be the sequence of coordinates for a given trajectory τ303such that c(τ)={c0, . . . , cT} and ct=(xt, yt, zt). Given an episode of length N, the transformation module118considers overlapping trajectories to be uniquely different such that c(τi) and c(τj) have the same probability

of being sampled, where τi={s0, . . . , sT}, τj={s1, . . . , sT+1}, and T<N. The transformation module118ensures that the AABA system106analyzes behavior (e.g., navigation/movement) without being biased by absolute location, by subtracting the initial Cartesian coordinate c0from each sample c(τ) so that each movement starts from the origin. As such, the output generated by the transformation module118is a sample distribution X309-1comprising the collection of vectors307-1generated for the artificial agent behavior information126-1and the sample distribution Y309-2comprising the collection of vectors307-1generated for the human behavior information126-2.

At310the behavior similarity determination module120of the AABA system106evaluates the behavior similarity of the agent behavior information126-1and the human behavior information126-2using the transformed behavior information represented by sample distribution X309-1and sample distribution Y309-2outputted by the transformation module118. As described above, the behavior similarity determination module120is configured based on a central hypothesis indicating that the similarity between the behavior of an artificial agent to that of a human counterpart is defined by the similarity between their respective distributions of trajectories303. In at least some embodiments, the behavior similarity determination module120takes as input the sample distribution X309-1and the sample distribution Y309-2, which were independently drawn from distributions Pθ305-1and P*305-2, respectively. As described above, Pθ305-1denotes the distribution of trajectories τ303-1induced by an artificial agent108following a decision policy πθ, while P*305-2denotes the distribution of trajectories τ303-2induced by a human counterpart. In at least some embodiments, the behavior similarity determination module120, for the purpose of evaluating the behavior similarity between P*305-1and Pθ305-1, is configured to evaluate the null hypothesis (H0) that these distributions are equal against the alternative hypothesis (H1) that they are different, as summarized below:

In at least some embodiments, the behavior similarity determination module120performs a two-sample hypothesis testing technique to evaluate the behavior126of the artificial agent(s)108and human counterpart(s).FIG.5illustrates, in flow chart form, an example method500for performing the two-sample hypothesis testing technique. The example method500is performed at block310ofFIG.3. The two-sample hypothesis testing technique, in at least some embodiments, is motivated by the following insight: if the null hypothesis is true, then any difference between P*305-1and Pθ305-2should be due to sampling error. As such, the behavior similarity determination module120implements one or more techniques to calculate a test statistic503for the purpose of evaluating the difference between sample distribution X309-1and sample distribution Y309-2. Examples of these techniques include the MMD and m out of n bootstrap resampling techniques described above. The behavior similarity determination module120then evaluates and compares this distribution of distances (e.g., MMD) in two settings, separated and pooled sample distributions, to generate behavior analysis results128including a similarity measurement128-1, such as a p-value507.

For example, at block502, the behavior similarity determination module120operates in the first setting to evaluate over separated sample distribution X309-1and sample distribution Y309-2. At block504, given that xiand yieach denote subsamples of size m that are independently drawn with replacement from sample distribution X309-1and sample distribution Y309-2, respectively, the behavior similarity determination module120forms a first distribution of MMD distances501-1by repeatedly recomputing MMDk[xi, yi] over S iterations where i∈{1, . . . , S}. This first distribution of distances501-1is denoted as δX,Y. At block506, the behavior similarity determination module120calculates the test statistic503, which is denoted as δ, according to EQ 4:

where quantile(δX,Y,α) returns the α-th quantile over the distribution δX,Yand α is a hyperparameter designed to control the sensitivity of the test, as described below. It should be understood that other techniques for generating the test statistic503are applicable as well. For example, confidence intervals can used to generate the test statistic503. In this example, given δX,Y, the mean and standard deviation is calculated so that the lower bound of a confidence interval defined by α is used to generate the test statistic503.

At block508, the behavior similarity determination module120operates in the second setting and combines sample distribution X309-1and sample distribution Y309-2to create a pooled sample distribution505denoted as Z. At block510, the behavior similarity determination module120performs an evaluation operation by forming a second distribution of MMD distances501-2. The similarity determination module120forms the second distribution of MMD distances501-2by repeatedly recomputing MMDk[xi,yi] over another S independently drawn samples where i∈{1, . . . , S}. However, in this second setting, xiand yiare both independently sampled from pooled distribution Z with replacement. This second distribution of distances501-2is denoted as δZ. At block512, the behavior similarity determination module120compares the distribution of distances501from the two settings to generate behavior analysis results128including a similarity measurement128-1quantifying how similar the artificial agent's behavior126-1is to the human counterpart's behavior126-2. In at least some embodiments, similarity measurement128-1is represented by a p-value507. The similarity determination module120, in at least some embodiments, generates the p-value507as the percentage of estimates δZgreater than the test statistic δ, as shown below in EQ 5, to evaluate the null hypothesis H0:

Referring toFIG.3, at block312the behavior similarity determination module120generates artificial agent behavior analysis results128including the similarity measurement128-1(e.g., p-value507).FIG.6illustrates one example of the artificial agent behavior analysis results128in table600form. In this example, the “T” column602represents the time horizon used in the evaluation, the “a” column604represents the quantile over baseline distribution. The “Human” column606represents the similarity measure between the behavior of two human counterparts. The “Agent 1” column608represents the similarity measure of a first artificial agent and the combined behavior of multiple human counterparts. The “Agent 2” column610represents the similarity measure of the behavior of a second artificial agent and the combined behavior of multiple human counterparts. Agent 1 and Agent 2, in this example, are trained/programmed. The percentages in the parentheses are the interquartile range (IQR) showing the degree of variance for the similarity measurements whereas the percentages outside of the parentheses are the medians over a given number of runs.

Referring toFIG.3, at block314, the AABA system106determines if the similarity measurement128-1indicates that the artificial agent108should be modified/reconfigured. For example, the AABA system106determines if the similarity measurement128-1indicates that the human-likeness of the artificial agent's behavior fails to satisfy a similarity threshold. If so, the process then exits at block318or, alternatively, returns to block302. Otherwise, at block316, the AABA system106modifies or reconfigures/programs the artificial agent108as described above with respect toFIG.1andFIG.2. For example, the artificial agent108is modified such that the human-likeness of the artificial agent's behavior is improved. Alternatively, the AABA system106instructs another system, such as the training system110, to perform the modification or reconfiguration of the artificial agent108. The process then exits at block318or, alternatively, returns to block302.

As described above, the p-value507, in at least some embodiments, is used by the AABA system106as a similarity measure128-1. In these embodiments, given that P* and Pθare the same distribution under the null hypothesis (H0), then the first distribution of MMD distances501-1as computed over the separated sample distribution X307-1and the sample distribution Y307-2should be the same as the second distribution of MMD distances501-2as computed over the pooled sample distribution Z505. Thus, when P*=Pθ, it follows that the resulting p-value507converges towards 1−α as S→∞. Furthermore, when P*≠Pθ, the AABA system106interprets the derived p-value507as a measure of closeness between distributions P* and Pθ. To demonstrate this, consider a series of experiments using a toy example where P* is distributed as a 128-dimensional standard Gaussian with zero mean and unit variance, which is denoted as(0,1). For each experiment, S=1000 iterations are run using subsample size m=100. Due to the inherent stochasticity of the two-sample hypothesis testing technique performed by the behavior similarity determination module120, each experiment is repeated 10 times and both the median p-value and the observed interquartile range (IQR) are reported.

First, the case where Pθ=P*=(0,1) using α=0.10 is considered. A median p-value of 88.5% with an IQR of 0.43% is observed. Next, to demonstrate how the p-value resulting from the two-sample hypothesis test performed by the behavior similarity determination module120is used as a measure of similarity between distribution Pθ305-1and distribution P*305-2, it is shown that p monotonically decreases as the distribution Pθ305-1is incrementally shifted by epsilon ϵ=0.02. Each Pθ=(ϵ,1) to P*=(0,1) is compared and the observed results are shown in the table700ofFIG.7. Furthermore, the distribution of the MMD distances501shifts with Pθ305-1. Note that when Pθ=(0.10,1), the distributions of δX,Yand δZare nearly separated. In at least some embodiments, the AABA system106controls the sensitivity of the two-sample hypothesis test by adjusting α. Intuitively, higher values of α yield a larger test statistic δ and, therefore, increase the sensitivity of the two-sample hypothesis test performed by the behavior similarity determination module120. As shown in the table700ofFIG.7, a more sensitive test yields lower p-values as the distributions diverge. Therefore, in cases where the behavior of multiple artificial agents108are being compared against the behavior of a human counterpart, a is adjustable to control for sensitivity and create a more informative comparison.

While the computational cost of estimating the MMD distance between two samples increases quadratically with the size of each sample, it also increases linearly with the dimensionality of the data. Therefore, in at least some embodiments, the dimensionality reduction module122of the AABA system106implements one or more dimensionality reduction techniques to reduce such computational costs by removing redundancies in high-dimensional data (e.g., images) and reducing the number of random variables under consideration (e.g., pixels) all while preserving the information needed to understand the original distribution. One example of a dimensionality reduction techniques is principal component analysis (PCA). To demonstrate the effects of PCA on the two-sample hypothesis test performed by the behavior similarity determination module120, consider a sample distribution D comprised of 22,600 fixed-length trajectories subsampled from 100 different episodes observed from 4 human counterparts. In this example, the dimensionality reduction module122linearly projects high-dimensional (HD) data into a low-dimensional (LD) space such that P128:2116→128where DHD∈2116, DLD∈128, and P128is the top 128 principal components which explain nearly 75% of variance in the data. In this example, reducing the dimensionality of the data significantly minimizes the impact of sample size on computational cost while maintaining the stability of the two-sample hypothesis test performed by the behavior similarity determination module120. It should be understood that other dimensionality techniques are applicable as well.

The techniques described herein are, in different embodiments, employed at any of a variety of processors or parallel processors (e.g., vector processors, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), non-scalar processors, highly parallel processors, artificial intelligence (AI) processors, inference engines, machine learning processors, other multithreaded processing units, and the like). Referring now toFIG.8, a block diagram of a processing system800, such as systems102-1to102-3is illustrated in accordance with some embodiments, configured with parallel processors. The processing system800includes a central processing unit (CPU)802and a graphics processing unit (GPU)804. In at least some embodiment, the CPU802, the GPU804, or both the CPU802and GPU804are configured to implement the AABA system106. The CPU802, in at least some embodiments, includes one or more single- or multi-core CPUs. In various embodiments, the GPU804includes any cooperating collection of hardware and or software that perform functions and computations associated with accelerating graphics processing tasks, data-parallel tasks, nested data-parallel tasks in an accelerated manner with respect to resources such as conventional CPUs, conventional graphics processing units (GPUs), and combinations thereof.

In the embodiment ofFIG.8, the CPU802and the GPU804are formed and combined on a single silicon die or package to provide a unified programming and execution environment. This environment enables the GPU804to be used as fluidly as the CPU802for some programming tasks. In other embodiments, the CPU802and the GPU804are formed separately and mounted on the same or different substrates. It should be appreciated that processing system800, in at least some embodiments, includes more or fewer components than illustrated inFIG.8. For example, the processing system800, in at least some embodiments, additionally includes one or more input interfaces, non-volatile storage, one or more output interfaces, network interfaces, and one or more displays or display interfaces.

As illustrated inFIG.8, the processing system800also includes a system memory806, an operating system808, a communications infrastructure810, and one or more applications812. Access to system memory806is managed by a memory controller (not shown) coupled to system memory806. For example, requests from the CPU802or other devices for reading from or for writing to system memory806are managed by the memory controller. In some embodiments, the one or more applications812include various programs or commands to perform computations that are also executed at the CPU802. The CPU802sends selected commands for processing at the GPU804. The operating system808and the communications infrastructure1010are discussed in greater detail below. The processing system800further includes a device driver814and a memory management unit, such as an input/output memory management unit (IOMMU)816. Components of processing system800are implemented as hardware, firmware, software, or any combination thereof. In some embodiments, the processing system800includes one or more software, hardware, and firmware components in addition to or different from those shown inFIG.8.

Within the processing system800, the system memory806includes non-persistent memory, such as dynamic random-access memory (not shown). In various embodiments, the system memory806stores processing logic instructions, constant values, variable values during execution of portions of applications or other processing logic, or other desired information. For example, in various embodiments, parts of control logic to perform one or more operations on CPU802reside within system memory806during execution of the respective portions of the operation by CPU802. During execution, respective applications, operating system functions, processing logic commands, and system software reside in system memory806. Control logic commands that are fundamental to operating system808generally reside in system memory806during execution. In some embodiments, other software commands (e.g., a set of instructions or commands used to implement a device driver814) also reside in system memory806during execution of processing system800.

The IOMMU816is a multi-context memory management unit. As used herein, context is considered the environment within which the kernels execute and the domain in which synchronization and memory management is defined. The context includes a set of devices, the memory accessible to those devices, the corresponding memory properties, and one or more command-queues used to schedule execution of a kernel(s) or operations on memory objects. The IOMMU816includes logic to perform virtual to physical address translation for memory page access for devices, such as the GPU804. In some embodiments, the IOMMU816also includes, or has access to, a translation lookaside buffer (TLB) (not shown). The TLB is implemented in a content addressable memory (CAM) to accelerate translation of logical (i.e., virtual) memory addresses to physical memory addresses for requests made by the GPU804for data in system memory806.

In various embodiments, the communications infrastructure810interconnects the components of the processing system800. Communications infrastructure810includes (not shown) one or more of a peripheral component interconnect (PCI) bus, extended PCI (PCI-E) bus, advanced microcontroller bus architecture (AMBA) bus, advanced graphics port (AGP), or other such communication infrastructure and interconnects. In some embodiments, communications infrastructure810also includes an Ethernet network or any other suitable physical communications infrastructure that satisfies an application's data transfer rate requirements. Communications infrastructure810also includes the functionality to interconnect components, including components of the processing system800.

A driver, such as device driver814, communicates with a device (e.g., GPU804) through an interconnect or the communications infrastructure810. When a calling program invokes a routine in the device driver814, the device driver814issues commands to the device. Once the device sends data back to the device driver814, the device driver814invokes routines in an original calling program. In general, device drivers are hardware-dependent and operating-system-specific to provide interrupt handling required for any necessary asynchronous time-dependent hardware interface. In some embodiments, a compiler818is embedded within device driver814. The compiler818compiles source code into program instructions as needed for execution by the processing system800. During such compilation, the compiler818applies transforms to program instructions at various phases of compilation. In other embodiments, the compiler818is a standalone application. In various embodiments, the device driver814controls operation of the GPU804by, for example, providing an application programming interface (API) to software (e.g., applications812) executing at the CPU802to access various functionality of the GPU804.

The CPU802includes (not shown) one or more of a control processor, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or digital signal processor (DSP). The CPU802executes at least a portion of the control logic that controls the operation of the processing system800. For example, in various embodiments, the CPU802executes the operating system808, the one or more applications812, and the device driver814. In some embodiments, the CPU802initiates and controls the execution of the one or more applications812by distributing the processing associated with one or more applications812across the CPU802and other processing resources, such as the GPU804.

The GPU804executes commands and programs for selected functions, such as graphics operations and other operations that are particularly suited for parallel processing. In general, GPU804is frequently used for executing graphics pipeline operations, such as pixel operations, geometric computations, and rendering an image to a display. In some embodiments, GPU804also executes compute processing operations (e.g., those operations unrelated to graphics such as video operations, physics simulations, computational fluid dynamics, etc.), based on commands or instructions received from the CPU802. For example, such commands include special instructions that are not typically defined in the instruction set architecture (ISA) of the GPU804. In some embodiments, the GPU804receives an image geometry representing a graphics image, along with one or more commands or instructions for rendering and displaying the image. In various embodiments, the image geometry corresponds to a representation of a two-dimensional (2D) or three-dimensional (3D) computerized graphics image.

In various embodiments, the GPU804includes one or more compute units, such as one or more processing cores820(illustrated as820-1and820-2) that include one or more single-instruction multiple-data (SIMD) units822(illustrated as822-1to822-4) that are each configured to execute a thread concurrently with execution of other threads in a wavefront by other SIMD units822, e.g., according to a SIMD execution model. The SIMD execution model is one in which multiple processing elements share a single program control flow unit and program counter and thus execute the same program but are able to execute that program with different data. The processing cores820are also referred to as shader cores or streaming multi-processors (SMXs). The number of processing cores820implemented in the GPU804is configurable. Each processing core820includes one or more processing elements such as scalar and or vector floating-point units, arithmetic and logic units (ALUs), and the like. In various embodiments, the processing cores820also include special-purpose processing units (not shown), such as inverse-square root units and sine/cosine units.

Each of the one or more processing cores820executes a respective instantiation of a particular work item to process incoming data, where the basic unit of execution in the one or more processing cores820is a work item (e.g., a thread). Each work item represents a single instantiation of, for example, a collection of parallel executions of a kernel invoked on a device by a command that is to be executed in parallel. A work item executes at one or more processing elements as part of a workgroup executing at a processing core820.

The GPU804issues and executes work-items, such as groups of threads executed simultaneously as a “wavefront”, on a single SIMD unit822. Wavefronts, in at least some embodiments, are interchangeably referred to as warps, vectors, or threads. In some embodiments, wavefronts include instances of parallel execution of a shader program, where each wavefront includes multiple work items that execute simultaneously on a single SIMD unit822in line with the SIMD paradigm (e.g., one instruction control unit executing the same stream of instructions with multiple data). A scheduler824is configured to perform operations related to scheduling various wavefronts on different processing cores820and SIMD units822and performing other operations to orchestrate various tasks on the GPU804.

To reduce latency associated with off-chip memory access, various GPU architectures include a memory cache hierarchy (not shown) including, for example, L1 cache and a local data share (LDS). The LDS is a high-speed, low-latency memory private to each processing core820. In some embodiments, the LDS is a full gather/scatter model so that a workgroup writes anywhere in an allocated space.

The parallelism afforded by the one or more processing cores820is suitable for graphics-related operations such as pixel value calculations, vertex transformations, tessellation, geometry shading operations, and other graphics operations. A graphics processing pipeline826accepts graphics processing commands from the CPU802and thus provides computation tasks to the one or more processing cores820for execution in parallel. Some graphics pipeline operations, such as pixel processing and other parallel computation operations, require that the same command stream or compute kernel be performed on streams or collections of input data elements. Respective instantiations of the same compute kernel are executed concurrently on multiple SIMD units822in the one or more processing cores820to process such data elements in parallel. As referred to herein, for example, a compute kernel is a function containing instructions declared in a program and executed on an accelerated processing device (APD) processing core820. This function is also referred to as a kernel, a shader, a shader program, or a program.

In at least some embodiments, the processing system800is a computer, laptop/notebook, mobile device, gaming device, wearable computing device, server, or any of various other types of computing systems or devices. It is noted that the number of components of the processing system800varies from embodiment to embodiment. In at least some embodiments, there is more or fewer of each component/subcomponent than the number shown inFIG.8. It is also noted that the processing system800, in at least some embodiments, includes other components not shown inFIG.8. Additionally, in other embodiments, the processing system800is structured in other ways than shown inFIG.8.

In some embodiments, the apparatus and techniques described above are implemented in a system including one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips). Electronic design automation (EDA) and computer-aided design (CAD) software tools, in at least some embodiments, are used in the design of the standard cells and the design and fabrication of IC devices implementing the standard cells. These design tools typically are represented as one or more software programs. The one or more software programs include code executable by a computer system to manipulate the computer system to operate on code representative of circuitry of one or more IC devices to perform at least a portion of a process to design or adapt a manufacturing system to fabricate the circuitry. This code, in at least some embodiments, includes instructions, data, or a combination of instructions and data. The software instructions representing a design tool or fabrication tool typically are stored in a computer-readable storage medium accessible to the computing system. Likewise, the code representative of one or more phases of the design or fabrication of an IC device, in at least some embodiments, is stored in and accessed from the same computer-readable storage medium or a different computer-readable storage medium.