INTERPRETABLE IMITATION LEARNING VIA PROTOTYPICAL OPTION DISCOVERY

A method for learning prototypical options for interpretable imitation learning is presented. The method includes initializing options by bottleneck state discovery, each of the options presented by an instance of trajectories generated by experts, applying segmentation embedding learning to extract features to represent current states in segmentations by dividing the trajectories into a set of segmentations, learning prototypical options for each segment of the set of segmentations to mimic expert policies by minimizing loss of a policy and projecting prototypes to the current states, training option policy with imitation learning techniques to learn a conditional policy, generating interpretable policies by comparing the current states in the segmentations to one or more prototypical option embeddings, and taking an action based on the interpretable policies generated.

BACKGROUND

Technical Field

The present invention relates to imitation learning and, more particularly, to methods and systems related to interpretable imitation learning via prototypical option discovery.

Description of the Related Art

Humans have the ability to compose options or skills to solve a complex problem. For example, to treat a COVID-19 patient with a critical condition, an intensive care unit (ICU) doctor needs to compose essential skills such as endotracheal intubation, chest-tube placement, and arterial and central venous catheterization. Discovering the compositional structures from experts' trajectories is beneficial to understand the experts' policy as well as learn a new policy.

SUMMARY

A method for learning prototypical options for interpretable imitation learning is presented. The method includes initializing options by bottleneck state discovery, each of the options presented by an instance of trajectories generated by experts, applying segmentation embedding learning to extract features to represent current states in segmentations by dividing the trajectories into a set of segmentations, learning prototypical options for each segment of the set of segmentations to mimic expert policies by minimizing loss of a policy and projecting prototypes to the current states, training option policy with imitation learning techniques to learn a conditional policy, generating interpretable policies by comparing the current states in the segmentations to one or more prototypical option embeddings, and taking an action based on the interpretable policies generated.

A non-transitory computer-readable storage medium comprising a computer-readable program for learning prototypical options for interpretable imitation learning is presented. The computer-readable program when executed on a computer causes the computer to perform the steps of initializing options by bottleneck state discovery, each of the options presented by an instance of trajectories generated by experts, applying segmentation embedding learning to extract features to represent current states in segmentations by dividing the trajectories into a set of segmentations, learning prototypical options for each segment of the set of segmentations to mimic expert policies by minimizing loss of a policy and projecting prototypes to the current states, training option policy with imitation learning techniques to learn a conditional policy, generating interpretable policies by comparing the current states in the segmentations to one or more prototypical option embeddings, and taking an action based on the interpretable policies generated.

A method for learning prototypical options for interpretable imitation learning is presented. The method includes dividing a task, by a processor, into a plurality of sub-tasks via a learning policy over options, learning, by the processor, different options to solve each of the plurality of sub-tasks by mimicking expert policy, and fine-tuning the learning policy to learn to take an action based on the task.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Imitation learning which mimics experts' behaviors is beneficial to finding meaningful structure or skills in the experts' demonstrations. Despite the superior performance of imitation learning models, they are usually considered as “black-boxes” which lack transparency, limiting their application in many decision-making scenarios, e.g., healthcare and finance. A variety of methods learn a hidden variable of the variation underlying expert demonstrations to construct the structure of expert policy and visualize the changes in the hidden variable. However, post-hoc explanations do not explain the reasoning process of how the model makes its decisions and can be incomplete or inaccurate in capturing the reasoning process of the original model. Therefore, it is often desirable to have models with built-in interpretability.

The exemplary embodiments address such issues by defining a form of interpretability in imitation learning that imitates human abstraction and explains its reasoning in a human-understanding manner. The exemplary methods employ prototype learning to discovery options for built-in interpretable imitation learning. Prototype learning, which drives from the study of human reasoning, is a form of case-based reasoning, which makes decisions by comparing new inputs with a few data instances (prototypes) in, e.g., image recognition, sequence classification, sequence segmentation, etc.

The exemplary methods discover prototypical options for interpretable imitation learning. The exemplary methods introduce a network architecture referred to as prototypical option discovery (IPOD). Each prototypical option is responsible for explaining a group of variable-length segments of the demonstration trajectories. To learn the prototypical options, IPOD first learns a policy to break the trajectories into a set of segmentations, which results in K groups of segments for the K prototypical options. IPOD uses LSTM with a soft-attention mechanism to derive segment embedding. For each group of segments, the exemplary methods learn a prototypical contextual policy to take action with states as well as the option embedding, which is determined based on centroids of the segment embedding, as inputs. In this way, the model is interpretable, in the sense that it has a transparent reasoning process when making decisions. For better interpretability, the exemplary methods define several criteria for constructing the prototypes, including option diversity and prediction accuracy.

The exemplary embodiments introduce an imitation learning framework that learns interpretable policy via prototypical options which include segmentation prototypes. The exemplary embodiments enable learning the prototypical option embedding by weighted segmentation for sparsity and learn the prototypical option's policy by driving the option-relevant information via option embedding. The goal is to learn a new policy it, which imitates the expert behavior by maximizing the likelihood of given demonstration trajectories. Thus, the behavior of an expert agent can be copied to accomplish a desired task.

Imitation learning refers to learning a policy that mimics the behavior of experts who demonstrate how to perform the given task. The behavior of the expert demonstrator is represented by trajectories τ=[s0, a0. . . , sT, aT], which is a sequence of state action pairs. Imitation learning has various approaches. One approach is behavior cloning (BC), which directly maps from the state to the action. This method usually learns a policy through standard supervised learning. BC does not perform any additional policy interactions with the learning environment, but it suffers from distributional drift. Another approach is inverse reinforcement learning (IRL), which learns a policy by recovering the reward function from demonstrations and with dense reward signals provided from the learned reward function. However, the learned policy is valid only while the learned reward function is valid. Yet another approach is adversarial imitation learning (AIL), which constrains the behavior of the agent to be approximately optimal with an unknown reward function without explicitly attempting to recover that reward function. However, both AIL and IRL require interacting with the environment for generating the agent's trajectory for comparison with the expert's trajectory. Recently, imitation learning with neural networks efficiently learns a desired behavior in complex environments. However, these methods are usually considered as “black-boxes,” which lack transparency. The exemplary methods introduce an interpretable imitation learning framework for more applications of imitation learning, e.g., healthcare, finance, etc.

An option is a generalization of an action (also known as a skill, sub-policy or a sub-goal). Formally, an option is a three-tuple that includes the start, end probability of an option and the policy of the option. Options offer great potential for mitigating the difficulty of solving complex Markov decision processes (MDPs) via temporally extended actions.

Interpretable modeling mainly falls into two categories, that is, intrinsic explanation which makes the model transparent by restricting the complexity, e.g., decision tree or case-based (prototype-based) model, and post-hoc explanation, which is achieved by analyzing the model after training, e.g., extracting the importance of states via attention and distilling a black-box policy into a simple structure policy. A set of post-hoc imitation learning was proposed for generating meaningful policy. However, the intrinsic explanation model is sometimes desirable since post-hoc explanations usually do not fit the original model precisely. Prototype learning, which draws conclusions for new inputs by comparing them with a few exemplary cases (e.g., prototypes) belongs to the intrinsic explanation method.

The options framework models skills as options, which is a closed-loop policy to solve the sub-tasks. For example, picking up an object, jumping, etc. are options, which require a user to take actions over a period of time. An option o includes the following components, that is, its initiation condition, Io(s), which determines whether o can be executed in state s, its termination condition, βo(s), which determines whether option execution must terminate in state s and its closed-loop control policy, πo(s), which maps state s to a low-level action a.

Prototype theory emerged in 1971 with the work of psychologist Eleanor Rosch, and it has been described as a “Copernican revolution” in the theory of categorization. In prototype theory, any given concept in any given language has a real-world example that best represents this concept. For instance, when asked to give an example of the concept of fruits, an apple is more frequently cited than, a durian. This theory claims that the presumed natural prototypes were central tendencies of the categories. Prototype theory has also been applied in machine learning, where a prototype is defined as a data instance that is representative of all the data. There are many approaches to find prototypes in the data. Any clustering algorithm that returns actual data points as cluster centers would qualify for selecting prototypes.

The exemplary embodiments introduce the formulation of the prototypical option, which is a kind of option that can be presented by an instance of the trajectories generated by the experts. A prototypical option o includes four components <Io, πo, βo, go>, that is, an intra-option policy πo:×→[0, 1], a termination condition βo:p→[0, 1], an initiation state set Io∈and an option prototype go.

Specifically, gois defined by sub-trajectories generated by the experts. Given the trajectories of the expert τ={s1, a1, . . . , sT, aT}, the prototypical option is a set of segments (g1, g2, . . . gK), where

A prototypical option <Io, πo, βo, go> is available in state stif and only if st∈Io. If the option is taken, then actions are selected according to πountil the option terminates according to βo. In a prototypical option, gois considered as a real-world example to explain the option.

Options discovery is based on the intuition that it would be easier to solve the long-horizon task from temporal abstraction, e.g., separate or divide the long-horizon task into a set of sub-tasks, and select different options to solve for each sub-task. This intuition informs the steps of the algorithm, that is, breaking or dividing the trajectories into a set of subtasks via learning a policy πhover options, learning (or discovering) options that could solve these sub-tasks by mimicking the expert' policy, and, once such options are learned, the exemplary embodiments fine-tune πhto learn to take an option based on the current task.

Formally, given the trajectories of the expert τ={s1, a1, . . . , sT, aT}, the goal is to first break or divide trajectories τ into M disjoint segments (g1, g2, . . . , gM), where

The exemplary embodiments leverage prototype learning to introduce an interpretable imitation learning framework by prototypical option discovery, where each prototypical option is responsible for explaining a group of variable-length segments of the demonstration trajectory. As presented inFIG. 2, I2L200addresses interpretable imitation learning tasks with steps to learn prototypical options <Io, πo, βo, go>. To learn the initial state set Ioand the termination condition βo, the exemplary methods learn a policy πh(o|s) over options to break or divide the trajectories into a set of segmentations, which results in K groups of segments for the K prototypical options. To learn the option prototype go, the exemplary methods map each segment into an option embedding

and cluster to them to find K central nodes as option prototypes go, o={1, . . . , K}. As for learning intra-option policy πo, the exemplary methods learn a prototypical contextual policy λ(a|s, o) to take action based on states, as well as the option embedding.

In options learning (Ioand βo) step, πh(o|s) first constructs a set of admissible options given by:(st)={oi|Ioi(st)=1∩βoi(st)=0, ∀oi∈}. Here the O(st) is updated according to the πh(o|s). IPOD200, inFIG. 2, determines the Ioi(st) and βoi(st) by the output of πh, e.g., ot, where if ot=1, Ioi(st)=1 and βoi(st)=0, otherwise Ioi(st)=0 and βoi(st)=1. An example of how the agent πh(o|s) selects an option is shown in structure100ofFIG. 1.

With regards to learning the policy over options, πh(o|s) is learned by choosing the admissible prototypical option. Since the exemplary methods utilize imitation learning to learn the intra-option policy, the reward of πh(o|s) is obtained by the selected option πowhich takes primitive actions and receives the reward signal. Thus, the reward of the option is the cumulative reward of the actions taken from a current time to the termination of the option:

where δ∈[0,T] is the time interval of the option t+δ is the termination of the option ot.

Given the transition (st, ot, rt:t+s) we update πh(o|s) taking option otat state staccording to policy gradient:

where the option-value Q(st, ot) refers to the expected rewards for an action ottaken in a given state st. Updating options to the policy over options, the above equations show how the exemplary methods can learn the policy πhover option and use it for selecting options. However, before learning πh, the exemplary methods must assign appropriate initial parameters to πh. The exemplary methods segment the trajectories by detecting the bottleneck states within the trajectories. Bottlenecks have been defined as those states which appear frequently on successful trajectories to a goal but not on unsuccessful ones or as nodes which allow for densely connected regions of the interaction graph to reach other such regions. Informally, bottleneck areas have been described as the border states of densely connected areas in the state space or as states that allow transitions to a different part of the environment. A more formal definition defines bottleneck areas as those states which are local maxima of betweenness, a measure of centrality on graphs, on a transition graph.

The exemplary methods extract all the states in the trajectories, and use density-based spatial clustering methods (e.g., DBSCAN) to automatically cluster the states into K groups. In the exemplary methods, each state group indicates one option's valid states (where Io(s)=1. That is, the initial πhwill take that option while it is in these states via behavior cloning.

In option prototype learning, the exemplary methods aim to learn the option prototype, which is a sub-trajectory or segment generated by the experts. Each option prototype is responsible for explaining a group of variable-length segments of the demonstration trajectory gmgenerated by πh. Thus, the exemplary methods first initialize K option prototype embedding ok∈n, k={1, 2, 3, . . . , K} vectors as learnable parameters. Next, the exemplary methods map each group of segments gm,kindividually into a low dimension embedding gm,kby classifying the segment into the corresponding option's category k. Meanwhile, the exemplary methods learn okby minimizing the distance between okand gm,k. Finally, the exemplary methods consider the segment which has the smallest distance with okas the option prototype of ok.

Regarding segmentation embedding learning, the exemplary methods aim to learn a meaningful latent space to represent the segments, where they are clustered (in L2-distance) around semantically similar prototypical options, and the clusters from different classes are well-separated.

To achieve this, the exemplary methods use a long short-term memory (LSTM) to learn the segment's representation

and the embeddings of prototypical option ok, where

vm=t indicates the current segment generated by πh. To force the segment

and the option prototypes to be in the same space, the exemplary methods minimize the distance between

and its closest prototype ok.

The optimization problem the exemplary methods aim to solve is:

The minimization ofembencourages each training segment to have some latent patch that is close to at least one prototypical option. These terms shape the latent space into a semantically meaningful clustering structure.

Regarding option prototype embedding learning (go), since the option prototype embeddings ok=1Kare representations in the latent space, they are not readily interpretable. For interpretability, the exemplary methods assign each prototypical option embedding okk=1Kwith their closest segment embedding g in the training set.

As for learning option prototype embedding, the exemplary methods leverage both supervised learning and imitation learning regarding the effectiveness and interpretability. The exemplary methods attempt to minimize the least square loss between g and ok, and prevent the learning of multiple similar prototypical options. The exemplary methods use a diversity regularization term that penalizes prototypical options that are close to each other. Meanwhile, the exemplary methods also consider the downstream task (e.g., imitation learning).

The full objective function of option learning is given as follows:

where the first term is for effectiveness, where an imitation learning objective function is conducted to learn the segment embeddings and option prototype embeddings to mimic expert's policy πE.IMloss(reproduced below) can be any imitation learning method, e.g., a behavior cloning loss or an adversarial imitation learning objective. The second term is for interpretability where an evidence regularization is used to encourage each prototypical option embedding to be as close to an encoded instance as possible. The third term is a diversity regularization term to learn diversified options, where dminis a threshold that classifies or determines whether two prototypes are close or not. dminis set to 1.0 in exemplary embodiments. λ1, λ2, λ3∈[0, 1] are the weights used to balance the three loss terms.

Regarding option policy learning π0, each option o maintains its own policy πo:s→at, which is parameterized by its own parameters θo. To reduce the parameter complexity, the exemplary methods propose a contextual policy ζθ(at|st, ok) to learn a conditional policy which is conditioned on both the state and the option, which is shared among all the options.

The exemplary methods train the option policy πθ(at|st, ok) via the traditional imitation learning algorithms defined asILloss, e.g., behavior cloning and adversarial imitation learning.

The goal of adversarial imitation learning is to minimize the JS divergence between trajectory distribution generated by the expert's policy and the option's policy.

Note that the exemplary methods use the same policy loss for both option prototypes and option policy, but the exemplary methods only optimize the parameters of option prototypes or option policy for each optimization step.

Regarding the full objective function, the loss minimized is:

where w1, w2, w3∈[0, 1] are hyper-parameters to balance the weights of the three kinds of loss. As for optimization, the exemplary methods first initialize K groups segments followed by iteratively optimizingoption+ILloss+emb.

Therefore, the exemplary embodiments introduce an interpretable imitation learning framework by discovering compositional structure which is called prototypical option discovery imitation learning (IPOD). IPOD constructs prototypical options which embed the skills of experts by an option embedding and an option policy via a prototype learning framework. IPOD generates interpretable agent policies by comparing the state segmentations to a few prototypical option embeddings followed by taking an action based on the option embedding. Unlike seeking a minimal subset of samples as prototypes that can serve as a distillation or condensed view of a data set, the exemplary model of the present invention uses a soft attention mechanism to derive prototypical option embedding from trajectory fragments. The exemplary methods also use the soft attention mechanism to create a bottleneck in the agent, forcing it to focus on option-relevant information.

FIG. 3is a block/flow diagram of an exemplary method300for employing the IPOD architecture ofFIG. 2, in accordance with embodiments of the present invention.

Prototypical option discovery for interpretable imitation learning (IPOD) proposes to learn prototypical options for interpretable imitation. Each prototypical option is responsible for explaining a group of variable-length segments of the demonstration trajectory. The exemplary methods model each group of segments by computing distances to prototypical option embedding, where prototypical option embedding is a latent variable summarizing the segments. The IPOD model includes the following learning phases.

At block303, option initialization takes place:

The IPOD first initializes the options by bottleneck state discovery methodology. Inspired by previous works on bottleneck state discovery, e.g., frequently visited states, the exemplary methods identify states that connect different densely connected regions in the state space. In order to discover such bottleneck states from expert demonstrations, the exemplary methods use the behavior cloning method with soft attention mechanism to obtain important states with large attention weights. The important states can then be found with DBSCAN clustering. The dense clusters derived from DBSCAN are used for option initialization.

At block305, the policy over options learning takes place:

A prototypical option o includes four components <Io, πo, βo, go>, an intra-option policy πo:×→[0, 1], a termination condition βo:→[0,1], an initiation state set I0∈, and its option prototype go. To select an option in state st, πh(o|s) first constructs a set of admissible options given by:

Here the(st) is updated according to the πh(o|s). IPOD determines the Ioi(st) and βoi(s_t) by the output of πh, i.e., ot, where if of=1, Ioi(st)=1 and βoi(st)=0. An example of how the agent πh(o|s) selects an option is shown above with respect to(st).

πh(o|s) is learned to choose the admissible prototypical option. Since the exemplary methods utilize imitation learning to learn the intra-option policy, the reward of πh(o|s) is obtained by the selected option πowhich takes primitive actions and receives the reward signal. Thus, the reward of the option is the cumulative reward of the actions taken from a current time to the termination of the option: r{t:t+δ}=rt+ . . . r{t+δ}, where δ∈[0, T] is the time interval of the option on-going, and t+δ is the termination of the option ot.

Given the transition (st, ot, rt:t+δ), the exemplary methods update πh(o|s) taking option otat state staccording to policy gradient:

where the option-value Q (st, ot) refers to the expected rewards for an action ottaken in a given state st.

At block307, prototypical option learning takes place:

In the second stage, the exemplary methods aim to learn the option prototype, which is a sub-trajectory or segment generated by the experts. Each option prototype is responsible for explaining a group of variable-length segments of the demonstration trajectory gm, generated by πh. Thus, the exemplary methods first initialize K option prototype embedding ok∈n, k={1, 2, 3, . . . , K} vectors as learnable parameters. Next, the exemplary methods map each group of segment gm,kindividually into a low-dimension embedding gm,kby classifying the segment into the corresponding option's category k. Meanwhile, the exemplary methods learn okby minimizing the distance between okand gm,k. Finally, the exemplary methods consider the segment which has the smallest distance with okas the option prototype of ok.

Regarding segmentation embedding learning, the exemplary methods aim to learn a meaningful latent space to represent the segments, where they are clustered (in L2-distance) around semantically similar prototypical options, and the clusters from different classes are well-separated.

To achieve this goal, the exemplary methods use an LSTM to learn the segment's representation gvm′:vm=fϕ(svm′:v_m) and the embeddings of prototypical option ok, where svm′:v_m, vm=t indicates the current segment generated by πh. To force the segment svm′:v_mand the option prototypes to be in the same space, the exemplary methods minimize the distance between gvm′vmand its closest prototype ok. The optimization problem to be solved is:

The minimization ofembencourages each training segment to have some latent patch that is close to at least one prototypical option. These terms shape the latent space into a semantically meaningful clustering structure.

At block309, prototypical option embedding learning takes place:

Since the option prototype embeddings ok=1Kare representations in the latent space, they are not readily interpretable. For interpretable, the exemplary methods propose to assign each prototypical option embedding ok=1Kwith their closest segment embedding g in the training set.

As for learning option prototype embedding, the exemplary methods leverage both supervised learning and imitation learning regarding effectiveness and interpretability. The exemplary methods try to minimize the least square loss between g and okto prevent learning multiple similar prototypical options. The exemplary methods use a diversity regularization term that penalizes prototypical options that are close to each other. Meanwhile, the exemplary methods also consider the downstream task (imitation learning).

The full objective function is:

where the first term is for effectiveness and where an imitation learning objective function is conducted to learn the segment embeddings and option prototype embeddings to mimic expert's policy πE. The second term is for interpretability where an evidence regularization is used to encourage each prototypical option embedding to be as close to an encoded instance as possible. The third term is a diversity regularization to learn diversified options and dminis a threshold that classifies whether two prototypes are close or not. The exemplary methods set dminto 1.0. λ1, λ2, λ3∈[0,1] are the weights used to balance the three loss terms.

At block311, option policy learning takes place:

Each option o maintains its own policy πo:s→at, which is parameterized by its own parameters θo. To reduce the parameter complexity, the exemplary methods propose a contextual policy πθ(at|st, ok) to learn a conditional policy which is conditioned on both the state and the option, which shares among all the options.

The exemplary methods train the option policy πo(at|st, ok) by traditional imitation learning algorithms, e.g., behavior cloning and adversarial imitation learning. The goal of behavior cloning is to mimic the action of the expert at each time step via supervised learning technical. The goal of adversarial imitation learning is to minimize the JS divergence between trajectory distribution generated by the expert's policy and the option's policy.

Note that the same policy loss is used for both option prototypes and option policy, but the exemplary methods only optimize the parameters of option prototypes or option policy for each optimization step. The exemplary methods can further train the option policy with imitation learning algorithms, e.g., behavior cloning and adversarial imitation learning. The goal of option policy learning is to mimic the segmentations of demonstrations from the experts.

FIG. 4is a block/flow diagram of an exemplary method for employing the option initialization, segmentation embedding learning, prototypical option learning, and option policy learning components ofFIG. 3, in accordance with embodiments of the present invention.

Imitation learning with neural networks efficiently learns a desired behavior in complex environments. However, these methods are usually considered as “black-boxes” which lack transparency, limiting their application in many decision-making scenarios. A variety of methods learn a hidden variable of the variation underlying expert demonstrations to construct the structure of expert policy and visualize the changes in the hidden variable. However, post-hoc explanations do not explain the reasoning process of how the model makes its decisions and can be incomplete or inaccurate in capturing the reasoning process of the original model. Therefore, it is often desirable to have models with built-in interpretability. The exemplary embodiments of the present invention define a form of interpretability in imitation learning that imitates human abstraction and explains its reasoning in a human-understanding manner. The exemplary methods enable prototype learning to discovery options for built-in interpretable imitation learning, which makes decisions by comparing the new inputs with a few data instances (prototypes).

Regarding the option initialization phase303:

At block401, attention mechanics and behavior cloning are utilized to extract the most important states considered while mimicking the expert's demonstration.

At block403, for bottleneck state discovery, DBSCAN is used on the extracted states and the states are automatically clustered into groups.

Regarding policy over options learning305:

At block411, imitation learning is utilized to learn the intra-option policy, where the reward is calculated by the cumulative rewards from the primitive actions.

Regarding prototypical option learning307:

At block421, prototypical options are learned via minimizing the loss of the policy and projecting the prototypes to observed states.

Regarding prototypical option embedding learning309:

At block431, prototypical options are learned via minimizing the loss of the policy and projecting the prototypes to observed states.

Regarding option policy learning311:

At block441, the option policy is trained with imitation learning algorithms, such as behavior cloning, inverse imitation learning and adversarial imitation learning.

In summary, the exemplary methods introduce a new architecture, that is, prototypical option discovery for interpretable imitation learning (IPOD). Each prototypical option includes a set of segmentation from experts' trajectories and is embedded by an option policy. The IPOD uses a soft attention mechanism to derive prototypical option embedding from its trajectory fragments. Given a demonstration of the expert, the model matches the segmentations from the demonstration to the learned prototypical options, and makes an action based on the learned prototypical option. The exemplary methods also use the soft attention mechanism to create a bottleneck in the agent, forcing the agent to focus on option-relevant information. In this way, the model is interpretable, in the sense that it has a transparent reasoning process when making decisions. For better interpretability, the exemplary methods define several criteria for constructing the prototypes, including option diversity and accuracy.

The IPOD considers the prototype learning to discovery options for built-in interpretable imitation learning in accordance with the following as illustrated inFIG. 2. Bottleneck state discovery segments the input trajectories into disjoint segments of variable length by, e.g., density-based clustering methods. Option projection includes representation learning of the segmentations in each cluster, and prototypical option embedding learning. Option refixation takes the low-level actions controlled through the prototypical option embedding and refines each group of segments by matching the segmentation embeddings to prototypical option embeddings.

FIG. 5is a block/flow diagram500of a practical application of the IPOD architecture, in accordance with embodiments of the present invention.

In one practical example, a patient502needs to receive medication504. Options are computed for indicating different levels of dosages of the medication504. The exemplary methods learn a prototypical contextual policy π(a|s, o) to take action based on states506. The IPOD architecture670is implemented to enable prototypical option visualization by executing a reasoning process555and evaluating policy performance557. I2L670, via the reasoning process555, can smoothly compose the different options by considering the variant states506of the patient502. In one instance, I2L670can chose the low-dosage option for the patient502. The results510(e.g., dosage options) can be provided or displayed on a user interface512handled by a user514.

FIG. 6is an exemplary processing system for GBL, in accordance with embodiments of the present invention.

The processing system includes at least one processor (CPU)604operatively coupled to other components via a system bus602. A GPU605, a cache606, a Read Only Memory (ROM)608, a Random Access Memory (RAM)610, an input/output (I/O) adapter620, a network adapter630, a user interface adapter640, and a display adapter650, are operatively coupled to the system bus602. Additionally, an interpretable imitation learning framework670can be employed to execute option initialization303, policy over options learning305, prototypical option learning307, prototypical option embedding learning309, and option policy learning311.

A storage device622is operatively coupled to system bus602by the I/O adapter620. The storage device622can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid-state magnetic device, and so forth.

A transceiver632is operatively coupled to system bus602by network adapter630.

User input devices642are operatively coupled to system bus602by user interface adapter640. The user input devices642can be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. Of course, other types of input devices can also be used, while maintaining the spirit of the present invention. The user input devices642can be the same type of user input device or different types of user input devices. The user input devices642are used to input and output information to and from the processing system.

A display device652is operatively coupled to system bus602by display adapter650.

FIG. 7is a block/flow diagram of an exemplary method for executing the IPOD architecture, in accordance with embodiments of the present invention.

At block701, initialize options by bottleneck state discovery, each of the options presented by an instance of trajectories generated by experts.

At block703, apply segmentation embedding learning to extract features to represent current states in segmentations by dividing the trajectories into a set of segmentations.

At block705, learn prototypical options for each segment of the set of segmentations to mimic expert policies by minimizing loss of a policy and projecting prototypes to the current states.

At block707, train option policy with imitation learning techniques to learn a conditional policy.

At block709, generate interpretable policies by comparing the current states in the segmentations to one or more prototypical option embeddings.

At block711, take an action based on the interpretable policies generated.

FIG. 8illustrates exemplary equations800for implementing the IPOD architecture, in accordance with embodiments of the present invention.

The equations include a loss function for segmentation embedding learning, an objective function, and policy losses.

As used herein, the terms “data,” “content,” “information” and similar terms can be used interchangeably to refer to data capable of being captured, transmitted, received, displayed and/or stored in accordance with various example embodiments. Thus, use of any such terms should not be taken to limit the spirit and scope of the disclosure. Further, where a computing device is described herein to receive data from another computing device, the data can be received directly from the another computing device or can be received indirectly via one or more intermediary computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and/or the like. Similarly, where a computing device is described herein to send data to another computing device, the data can be sent directly to the another computing device or can be sent indirectly via one or more intermediary computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and/or the like.