Patent Publication Number: US-2021192391-A1

Title: Method and apparatus for performing learning from demonstrations, particularly imitation learning

Description:
CROSS REFERENCE 
     The present application claims the benefit under 35 U.S.C. § 119 of European Patent Application No. EP 19218664.1 filed on Dec. 20, 2019, which is expressly incorporated herein by reference in its entirety. 
     FIELD 
     Exemplary embodiments of the present invention relate to a method for performing Learning from Demonstrations, LfD, particularly Imitation Learning, IL, based on data associated with a first domain. 
     Further exemplary embodiments of the present invention relate to an apparatus for performing Learning from Demonstrations, particularly Imitation Learning, based on data associated with a first domain. 
     SUMMARY 
     Exemplary preferred embodiments of the present invention relate to a method, preferably a computer-implemented method, for performing Learning from Demonstrations, LfD, particularly Imitation Learning, based on data associated with a first domain, particularly a source domain, said method comprising: determining first data characterizing a demonstrator (particularly a behavior such as e.g. a movement of said demonstrator) of said first domain, wherein particularly said first data characterizes sensor data of said demonstrator and/or sensor data of at least one spectator observing said demonstrator, determining first knowledge from said first domain based on said first data, transferring at least a part of said first knowledge to a second domain, particularly a target domain. This enables to employ said at least part of said first knowledge in a second domain. 
     According to further preferred embodiments of the present invention, this approach may, e.g., be used in the field of “Learning from Demonstrations” (LfD), e.g., to address at least one of the tasks: (1) inferring an effect of a selected action on outcome given observation, and (2) imitation learning. 
     According to further preferred embodiments, major issues in Learning from Demonstration may arise when the sensors that record the demonstrator (or known as experts) differ from those available to an (artificial intelligence, AI) agent that is to be trained. For instance, according to further preferred embodiments, e.g., for the development of self-driving cars, drones may be deployed to fly over highways to record comparatively large amounts of demonstrations by human-driven cars. According to inventors&#39; analysis, in such drone recordings, some crucial variables for applying LfD techniques may either completely be missing, e.g., indicator lights of the observed cars, or they may be more noisy than a human or sensor can observe from within a car, in contrast to the perspective of the drone(s). 
     Further, according to inventors&#39; analysis, ignoring such issues, or addressing them in a naive way, may result in significantly wrong conclusions about demonstrator&#39;s behavior and the demonstrator&#39;s actions&#39; effects on the environment. As a simple example according to further embodiments, assume one wants to use highway drone data, to learn how an acceleration action A of a “demonstrator car” affects a lane changing behavior Z of a “lead car” in front of it on a slower (e.g., right) lane. 
     According to further preferred embodiments of the present invention, slightly simplifying the reality, assume the indicator light of the lead car serves as a perfect coordination device: whenever it is on, it will result in (1) the demonstrator car decelerating and (2) the lead car changing lane to the fast lane. Now assume one just uses the variables recorded in the drone data of said drone(s), where the indicator light is not contained, estimate P(Z|A) (probability of changing lane given the acceleration) from it. This may lead to the conclusion that an agent in the place of the demonstrator can arbitrarily chose any acceleration or deceleration action, and the lead car will perfectly adapt Z and only change lane when agent decelerates—which in practice can lead to crashes. 
     To at least partly mitigate these disadvantages, the principle according to the example embodiments is proposed. According to further preferred embodiments of the present invention, said steps of determining first data characterizing said demonstrator of said first domain (i.e., characterizing a behavior such as, e.g., a movement of said demonstrator), determining first knowledge from said first domain based on said first data, transferring at least a part of said first knowledge to a second domain enable learning from demonstrations (LfD) under sensor-shift, i.e., when a) the sensors of the demonstrator, and/or b) the sensors that are used to observe the demonstrator, and/or c) the sensors of an AI agent that is to be trained depending on said demonstrator are different. 
     According to further preferred embodiments of the present invention, said first data may comprise sensor data of said demonstrator and/or of at least one observer configured to at least temporarily observe said demonstrator, and/or data derived from said sensor data of said demonstrator and/or of at least one observer configured to at least temporarily observe said demonstrator. 
     According to further preferred embodiments of the present invention, sensor characteristics of a spectator in the first, i.e., source, domain, and/or sensor characteristics of a target agent in the second, i.e., target, domain, i.e., P_S(Y_S|X) and P_T(Y_T|X) may (preferably additionally) be used. 
     According to further preferred embodiments of the present invention, which are explained in detail further below, methods and techniques are provided that enable to use causal models to, preferably rigorously, analyze, particularly on a population-level, to what extent the relevant underlying mechanisms (the decision-effect and the demonstrator policy) can be identified and transferred from the available observations. Furthermore, further preferred embodiments of the present invention provide algorithms to determine, particularly calculate, them. 
     According to further preferred embodiments of the present invention, proxy methods are introduced which may—at least in some cases—be easier to calculate, and/or to estimate from finite data and/or to interpret than the exact solutions, alongside theoretical bounds on their closeness to the exact ones. 
     According to further preferred embodiments of the present invention, the first domain may also be denoted as “source domain”, wherein said demonstrator (e.g., demonstrator car according to the abovementioned example) acts. 
     According to further preferred embodiments of the present invention, the second domain may also be denoted as “target domain”, wherein a target agent (referred to as “AI agent” in the abovementioned example) observes (e.g., to perform LfD and/or to be trained using LfD) and acts. 
     According to further preferred embodiments of the present invention, the term “domain” may denote a, preferably complete, causal structure of environment and/or sensors and/or the respective agent(s). 
     Further preferred embodiments of the present invention further comprise: modeling the first (e.g., source) domain by means of a, preferably causal, directed acyclic graph, DAG, and/or modeling the second (e.g., target) domain by means of a, preferably causal, directed acyclic graph. 
     According to further preferred embodiments of the present invention, the following variables may be used, e.g. within said DAGs: variable “X” may characterize a state of the system, variable “A” may characterize an action of an agent, and variable “Z” may characterize (i.e., stand for) an outcome (e.g., an abstract variable that could e.g. be, say, the state of cars in a the next time instance given a state and an action). 
     According to further preferred embodiments of the present invention, the following variables may be used, especially to characterize observations: the variable “E D ” may characterize the demonstrator&#39;s input, e.g., generated by the demonstrator&#39;s sensor(s), the variable “Y S ” may characterize the AI agent&#39;s observation of the state of the source system (e.g., drone data from the high-way, in the context of the aforementioned example), and in the target domain, the variable “Y T ” may characterize an input to the target agent measured by its sensors. 
     According to further preferred embodiments of the present invention, let the distributions over variables in the source domain and the target domain (e.g. P(Z)) be denoted by subscript “S” and “T”, respectively (e.g., P S (Z) and P T (Z)). According to further preferred embodiments, let π D (a|Y D ) denote a policy of the demonstrator (also denoted as “expert”), and π T (a|Y T ) denote a policy of the target agent. 
     Further preferred embodiments of the present invention relate to a method of designing a target agent that observes and successfully acts in the target domain, e.g., based on what is known and/or what may be derived from the source domain (e.g., from observing the demonstrator) and its relation to the target domain. 
     According to further preferred embodiments of the present invention, said method further comprises: determining, particularly inferring, an effect of an action to an outcome in the second domain, particularly conditional on an observation in said second domain. 
     According to further preferred embodiments of the present invention, said method further comprises: providing a utility function u(Z) associated with said outcome, and, optionally, determining a first action by maximizing said utility function u with respect to an action a, particularly based on said observation, wherein said optional step of determining may, e.g., be performed based on the equation E[u(Z)|do(a), Y T ], wherein E[ ] is an expected value. 
     According to further preferred embodiments of the present invention, said method further comprises: determining, particularly inferring, a conditional distribution over actions a given an or said observation in the second domain, preferably such that a target agent associated with said second domain behaves similar to a or said demonstrator of the first domain. 
     According to further preferred embodiments of the present invention, said method further comprises: a) using, particularly characterizing one or more aspects of at least one of said DAGs with, the equation, 
     
       
         
           
             
               
                 p 
                 S 
               
                
               
                 ( 
                 
                   z 
                   , 
                   a 
                   , 
                   
                     y 
                     S 
                   
                 
                 ) 
               
             
             = 
             
               
                 
                   x 
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         y 
                         S 
                       
                        
                       x 
                     
                     ) 
                   
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       z 
                       , 
                       a 
                       , 
                       x 
                     
                     ) 
                   
                 
               
               = 
               
                 
                   
                     x 
                     , 
                     
                       y 
                       D 
                     
                   
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         y 
                         S 
                       
                        
                       x 
                     
                     ) 
                   
                 
                  
                 
                   p 
                    
                   
                     ( 
                     
                       
                         z 
                          
                         a 
                       
                       , 
                       x 
                     
                     ) 
                   
                 
                  
                 
                   
                     π 
                     D 
                   
                    
                   
                     ( 
                     
                       a 
                        
                       
                         y 
                         D 
                       
                     
                     ) 
                   
                 
                  
                 
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           y 
                           D 
                         
                         , 
                         x 
                       
                       ) 
                     
                   
                   . 
                 
               
             
           
         
       
     
     and/or b) using, particularly characterizing one or more aspects of at least one of said DAGs with, the equation 
     
       
         
           
             
               
                 
                   
                     [ 
                     
                       
                         
                           
                             P 
                              
                             
                               ( 
                               
                                 z 
                                 , 
                                 a 
                                 , 
                                 
                                   y 
                                   1 
                                 
                               
                               ) 
                             
                           
                         
                       
                       
                         
                           ⋮ 
                         
                       
                       
                         
                           
                             P 
                              
                             
                               ( 
                               
                                 z 
                                 , 
                                 a 
                                 , 
                                 
                                   y 
                                   m 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                     ] 
                   
                   = 
                   
                     
                       [ 
                       
                         
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   
                                     y 
                                     1 
                                   
                                    
                                   
                                     x 
                                     1 
                                   
                                 
                                 ) 
                               
                             
                           
                           
                             … 
                           
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   
                                     y 
                                     1 
                                   
                                    
                                   
                                     x 
                                      
                                   
                                 
                                 ) 
                               
                             
                           
                         
                         
                           
                             ⋮ 
                           
                           
                             
                                 
                             
                           
                           
                             ⋮ 
                           
                         
                         
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   
                                     y 
                                     m 
                                   
                                    
                                   
                                     x 
                                     1 
                                   
                                 
                                 ) 
                               
                             
                           
                           
                             … 
                           
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   
                                     y 
                                     m 
                                   
                                    
                                   
                                     x 
                                      
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       ] 
                     
                      
                     
                       [ 
                       
                         
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   z 
                                   , 
                                   a 
                                   , 
                                   
                                     x 
                                     1 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                         
                           
                             ⋮ 
                           
                         
                         
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   z 
                                   , 
                                   a 
                                   , 
                                   
                                     x 
                                      
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   [ 
                   
                     equation 
                      
                     
                         
                     
                      
                     1 
                   
                   ] 
                 
               
             
           
         
       
     
     (this is a matrix form of the preceding equation according to further preferred embodiments, where all variables are discrete), preferably for all values of z and/or all values of a. 
     In the above equation P S (z, a, y S ) denotes a joint distribution of outcome, action, and the observation in the source domain, P S (y S |x) is a conditional distribution of the observation in the source domain given the state, and P S (z, a, x) is a joint distribution of outcome, action, and state, respectively. P(z|a, x) is a conditional distribution of the outcome given action and state. 
     According to further preferred embodiments of the present invention, said method further comprises: using, particularly for said step of determining, the equation 
     
       
         
           
             
               
                 
                   
                     
                       
                         p 
                         ~ 
                       
                        
                       
                         ( 
                         
                           
                             
                               Z 
                                
                               X 
                             
                             = 
                             x 
                           
                           , 
                           A 
                         
                         ) 
                       
                     
                     := 
                     
                       
                         ∑ 
                         y 
                       
                        
                       
                         
                           
                             p 
                             S 
                           
                            
                           
                             ( 
                             
                               
                                 
                                   Z 
                                    
                                   
                                     Y 
                                     S 
                                   
                                 
                                 = 
                                 y 
                               
                               , 
                               A 
                             
                             ) 
                           
                         
                          
                         
                           p 
                            
                           
                             ( 
                             
                               
                                 Y 
                                 S 
                               
                               = 
                               
                                 y 
                                  
                                 s 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   [ 
                   
                     equation 
                      
                     
                         
                     
                      
                     2 
                   
                   ] 
                 
               
             
           
         
       
     
     wherein {tilde over (p)}(Z|X=x, A) characterizes a proxy for the conditional distribution of outcome given action and state (particularly an average-based action-effect proxy), wherein p S (Z|Y S =y, A) characterizes a conditional distribution of the outcome given the action and the observation in the source domain, and wherein p(Y S =y|x) characterizes a conditional distribution of the observation in the source domain given the state. 
     According to further preferred embodiments of the present invention, the [equation 2] characterizes or represents an average-based decision-effect proxy of p(z|X, A), defined, preferably only, based on known aspects, i.e., from the source domain. According to further preferred embodiments of the present invention, a deviation between the average-based proxy ([equation 2]) and the ground truth that it approximates can be bounded in the following way: D(P S (Z|X,A)∥{tilde over (P)}(Z|X,A))≤I S (X:Z|A,Y S ). In this inequality, D(⋅∥⋅) denotes the Kullback-Leibler (KL) divergence and I S (X:Z|A,Y S ) is a conditional mutual information between the state and the outcome given the action and the observation in the source domain. 
     According to further preferred embodiments of the present invention, said method further comprises: using at least one of the following equations, particularly for said step of determining, particularly inferring, said conditional distribution over actions a: a) 
     
       
         
           
             
               
                 
                   
                     
                       
                         π 
                         ~ 
                       
                       T 
                       
                         ( 
                         1 
                         ) 
                       
                     
                      
                     
                       ( 
                       
                         
                           a 
                            
                           
                             Y 
                             T 
                           
                         
                         = 
                         y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         y 
                         ′ 
                       
                     
                      
                     
                       
                         p 
                         S 
                       
                        
                       
                         ( 
                         
                           
                             a 
                              
                             
                               Y 
                               S 
                             
                           
                           = 
                           
                             y 
                             ′ 
                           
                         
                         ) 
                       
                     
                      
                     
                       
                         p 
                         S 
                       
                        
                       
                         ( 
                         
                           
                             Y 
                             S 
                           
                           = 
                           
                             
                               
                                 y 
                                 ′ 
                               
                                
                               
                                 Y 
                                 D 
                               
                             
                             = 
                             y 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   
                     
                       [ 
                       
                         equation 
                          
                         
                             
                         
                          
                         3 
                          
                         a 
                       
                       ] 
                     
                     , 
                     b 
                   
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         π 
                         ~ 
                       
                       T 
                       
                         ( 
                         2 
                         ) 
                       
                     
                      
                     
                       ( 
                       
                         a 
                          
                         
                           y 
                           T 
                         
                       
                       ) 
                     
                   
                   ∝ 
                   
                     exp 
                      
                     
                       ( 
                       
                         
                           
                             y 
                             S 
                           
                         
                          
                         
                           p 
                            
                           
                             ( 
                             
                               
                                 y 
                                 S 
                               
                                
                               
                                 y 
                                 T 
                               
                             
                             ) 
                           
                         
                          
                         log 
                          
                         
                             
                         
                          
                         
                           
                             p 
                             S 
                           
                            
                           
                             ( 
                             
                               a 
                                
                               
                                 y 
                                 S 
                               
                             
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   
                     
                       [ 
                       
                         equation 
                          
                         
                             
                         
                          
                         3 
                          
                         b 
                       
                       ] 
                     
                     , 
                     c 
                   
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         π 
                         ~ 
                       
                       T 
                       
                         ( 
                         3 
                         ) 
                       
                     
                      
                     
                       ( 
                       
                         a 
                          
                         
                           y 
                           T 
                         
                       
                       ) 
                     
                   
                   ∝ 
                   
                     exp 
                      
                     
                       ( 
                       
                         
                           x 
                         
                          
                         
                           p 
                            
                           
                             ( 
                             
                               x 
                                
                               
                                 y 
                                 T 
                               
                             
                             ) 
                           
                         
                          
                         log 
                          
                         
                           
                             p 
                             ~ 
                           
                            
                           
                             ( 
                             
                               a 
                                
                               x 
                             
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   [ 
                   
                     equation 
                      
                     
                         
                     
                      
                     3 
                      
                     c 
                   
                   ] 
                 
               
             
           
         
       
     
     According to further preferred embodiments of the present invention, when using [equation 3a], a deviation from its true distribution is bounded by the following result: D(π D ∥π T   (1) )≤I S (A;Y D |Y S ). 
     According to further preferred embodiments, when using [equation 3c], the expression {tilde over (p)}(a|x) is defined by 
     
       
         
           
             
               
                 p 
                 ~ 
               
                
               
                 ( 
                 
                   a 
                    
                   x 
                 
                 ) 
               
             
             := 
             
               
                 y 
               
                
               
                 
                   p 
                   S 
                 
                  
                 
                   ( 
                   
                     
                       a 
                        
                       
                         Y 
                         S 
                       
                     
                     = 
                     y 
                   
                   ) 
                 
               
                
               
                 
                   p 
                   S 
                 
                  
                 
                   ( 
                   
                     
                       Y 
                       S 
                     
                     = 
                     
                       y 
                        
                       x 
                     
                   
                   ) 
                 
               
             
           
         
       
     
     According to further preferred embodiments, said method further comprises: a) using, if a or said observation in said second domain is the same as an expert&#39;s observation (i.e., Y T =Y D ), the equation 
     
       
         
           
             
               
                 
                   
                     π 
                     ~ 
                   
                   T 
                   
                     ( 
                     1 
                     ) 
                   
                 
                  
                 
                   ( 
                   
                     
                       a 
                        
                       
                         Y 
                         T 
                       
                     
                     = 
                     y 
                   
                   ) 
                 
               
               = 
               
                 
                   
                     y 
                     ′ 
                   
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         a 
                          
                         
                           Y 
                           S 
                         
                       
                       = 
                       
                         y 
                         ′ 
                       
                     
                     ) 
                   
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         Y 
                         S 
                       
                       = 
                       
                         
                           
                             y 
                             ′ 
                           
                            
                           
                             Y 
                             D 
                           
                         
                         = 
                         y 
                       
                     
                     ) 
                   
                 
               
             
             , 
           
         
       
     
     and/or b) using, if demonstrator&#39;s sensors and a target agent&#39;s sensors in the first domain are the same, but if the target agent&#39;s sensors in the first domain are different from the target agent&#39;s sensors in the second domain (i.e., Y T ≠Y D =Y S ), the equation 
     
       
         
           
             
               
                 
                   π 
                   ~ 
                 
                 T 
                 
                   ( 
                   2 
                   ) 
                 
               
                
               
                 ( 
                 
                   a 
                    
                   
                     y 
                     T 
                   
                 
                 ) 
               
             
             ∝ 
             
               exp 
                
               
                 ( 
                 
                   
                     
                       y 
                       S 
                     
                   
                    
                   p 
                    
                   
                     ( 
                     
                       
                         y 
                         S 
                       
                        
                       
                         y 
                         T 
                       
                     
                     ) 
                   
                    
                   log 
                    
                   
                       
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         a 
                          
                         
                           y 
                           S 
                         
                       
                       ) 
                     
                   
                 
                 ) 
               
             
           
         
       
     
     and/or c) using, if the demonstrator&#39;s sensors and the target agent&#39;s sensors in the first domain and the target agent&#39;s sensors in the second domain are each different from each other (i.e., Y T ≠Y D ≠Y S ), the equation 
     
       
         
           
             
               
                 
                   
                     π 
                     ~ 
                   
                   T 
                   
                     ( 
                     3 
                     ) 
                   
                 
                  
                 
                   ( 
                   
                     a 
                      
                     
                       y 
                       T 
                     
                   
                   ) 
                 
               
               ∝ 
               
                 exp 
                  
                 
                   ( 
                   
                     
                       x 
                     
                      
                     p 
                      
                     
                       ( 
                       
                         x 
                          
                         
                           y 
                           T 
                         
                       
                       ) 
                     
                      
                     log 
                      
                     
                       
                         p 
                         ~ 
                       
                        
                       
                         ( 
                         
                           a 
                            
                           x 
                         
                         ) 
                       
                     
                   
                   ) 
                 
               
             
             , 
             
               
 
             
              
             
               
                 wherein 
                  
                 
                     
                 
                  
                 
                   
                     p 
                     ~ 
                   
                    
                   
                     ( 
                     
                       a 
                        
                       x 
                     
                     ) 
                   
                 
               
               := 
               
                 
                   y 
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         a 
                          
                         
                           Y 
                           S 
                         
                       
                       = 
                       y 
                     
                     ) 
                   
                 
                  
                 
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           Y 
                           S 
                         
                         = 
                         
                           y 
                            
                           x 
                         
                       
                       ) 
                     
                   
                   . 
                 
               
             
           
         
       
     
     According to further embodiments of the present invention, in the above equations, variables with tilde (“˜”) represent the proxies. According to further embodiments, P(y S |y T ) denotes a conditional distribution of the observation in the source domain given the observation in the target domain. The other variables are explained above. 
     Further preferred embodiments of the present relate to an apparatus configured to perform the example method(s) described above. 
     Further preferred embodiments of the present invention relate to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method(s) according to the embodiments. 
     Further preferred embodiments of the present invention relate to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the example embodiments. 
     Further preferred embodiments relate to a data carrier signal carrying the computer program according to the example embodiments. 
     Further preferred embodiments relate to a use of the method according to the embodiments and/or of the apparatus according to the embodiments and/or of the computer program according to the embodiments for at least one of: a) training a machine learning system, b) determining and/or providing training data, particularly for a machine learning system, c) learning from demonstrations, particularly under sensor-shift, i.e., when the sensors of the demonstrator and/or the sensors that are used to observe the demonstrator, and/or the sensors of the AI agent that is to be trained (or respective data as can be obtained from these sensors, respectively) are different. 
     According to further preferred embodiments of the present invention, the principle according to the embodiments can be used to receive sensor signals (e.g., of the demonstrator and/or of at least one spectator or observer observing said demonstrator), e.g., from at least one sensor such as, e.g., video, radar, LiDAR, ultrasonic, motion, and the like, to name a few, wherein, e.g., at least parts of said sensor signals may be processed as said data associated with the first domain according to preferred embodiments. 
     According to further preferred embodiments of the present invention, the principle according to the embodiments can be used to compute one or more control signals for controlling a physical system, like e.g. a computer-controlled machine, like a robot, a vehicle, a manufacturing machine, a personal assistant or an access control system or the like, and/or for providing a control signal for controlling said physical system, particularly a technical system, particularly a vehicle. 
     Further preferred embodiments of the present invention may comprise at least one of the following steps: a) analyzing and/or processing the sensor data, b) incorporating specifications of sensors in both the source domain (first domain) and the target (second) domain(s), c) transfer extracted knowledge from the source domain to the target domain. 
     According to further preferred embodiments of the present invention, the principle according to the embodiments can be used to train a machine learning system that can be used for, e.g., the above applications. According to further preferred embodiments, such machine learning system may e.g. be provided using the apparatus according to the embodiments. 
     According to further preferred embodiments of the present invention, the principle according to the embodiments can be used to generate training data for training a machine learning system that can, e.g., be used for the above applications. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Some exemplary embodiments will now be described with reference to the accompanying figures. 
         FIG. 1  schematically depicts a simplified block diagram according to preferred embodiments of the present invention. 
         FIG. 2A  schematically depicts a simplified flow chart of a method according to further preferred embodiments of the present invention. 
         FIG. 2B  schematically depicts a simplified flow chart of a method according to further preferred embodiments of the present invention. 
         FIG. 2C  schematically depicts a simplified flow chart of a method according to further preferred embodiments of the present invention. 
         FIG. 3  schematically depicts two graphs according to further preferred embodiments of the present invention. 
         FIG. 4  schematically depicts a simplified block diagram of an apparatus according to further preferred embodiments of the present invention. 
         FIG. 5A  schematically depicts a simplified block diagram according to further preferred embodiments of the present invention. 
         FIG. 5B  schematically depicts a simplified block diagram according to further preferred embodiments of the present invention. 
         FIG. 6  schematically depicts a scenario according to further preferred embodiments of the present invention. 
         FIG. 7A  schematically depicts a simplified flow chart of a method according to further preferred embodiments of the present invention. 
         FIG. 7B  schematically depicts a simplified flow chart of a method according to further preferred embodiments of the present invention. 
         FIG. 8  schematically depicts a simplified flow chart of a method according to further preferred embodiments of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS 
       FIG. 1  schematically depicts a simplified block diagram according to preferred embodiments. 
     Depicted is a first domain D_ 1  and a second domain D_ 2 . According to further preferred embodiments, the first domain D_ 1  may also be denoted as “source domain”, wherein a demonstrator  10 , e.g., a demonstrator car  10  as exemplarily depicted by  FIG. 6 , acts. 
     According to further preferred embodiments, the second domain D_ 2  may also be denoted as “target domain”, wherein a target agent  20  (or “AI agent”) may observe (e.g., to perform LfD) and act. 
     According to further preferred embodiments, the term “domain” may denote a, preferably complete, causal structure of environment, sensors, and the respective agent(s). Hence, according to further preferred embodiments, the first domain D_ 1  and/or the second domain D_ 2  of  FIG. 1  may represent a, preferably complete, causal structure of environment, sensors, and the respective agent(s)  10 ,  20 . 
     Exemplary preferred embodiments, cf.  FIG. 2A , relate to a method, preferably a computer-implemented method, of processing data associated with a first domain D_ 1  ( FIG. 1 ), particularly for performing Learning from Demonstrations, LfD, particularly Imitation Learning, based on said data associated with said first domain, particularly a source domain, said method comprising: determining  100  ( FIG. 2A ) first data Dat_ 1  characterizing a demonstrator  10  of said first domain D_ 1 , wherein particularly said first data Dat_ 1  characterizes sensor data of said demonstrator  10  and/or sensor data of at least one spectator (“observer”) observing said demonstrator  10 , determining  110  first knowledge KN_ 1  from said first domain D_ 1  based on said first data Dat_ 1 , transferring  120  at least a part KN_ 1 ′ of said first knowledge KN_ 1  to a second domain D_ 2 , which e.g. is a target domain. This enables to employ said at least part KN_ 1 ′ of said first knowledge KN_ 1  to a second domain D_ 2 , which is e.g. associated with an agent to be trained, e.g. by using an LfD or IL technique. 
     According to further preferred embodiments, the approach exemplarily depicted by  FIG. 1, 2A  may e.g. be used in the field of “Learning from Demonstrations” (LfD), e.g. to address at least one of the tasks: (1) inferring an effect of a selected action on outcome given observation, and (2) imitation learning. 
     According to further preferred embodiments, major issues in LfD may arise when the sensors that record the demonstrator (“expert”)  10  ( FIG. 1 ) differ from those available to an (artificial intelligence, AI) agent  20  that is to be trained. For instance, according to further preferred embodiments, e.g. for the development of self-driving cars, drones DR ( FIG. 6 ) may be deployed to fly over highways HW to record comparatively large amounts of demonstrations, e.g. provided by human-driven cars  2 . According to inventors&#39; analysis, in such drone recordings, some crucial variables for applying LfD techniques may either completely be missing, e.g., indicator lights IL of the observed cars  2 , or they may be more noisy than a human or sensor can observe from within a car, in contrast to the perspective of the drone(s) DR. 
     Further, according to inventors&#39; analysis, ignoring such issues, or addressing them in a naive way, may result in significantly wrong conclusions about demonstrator&#39;s behavior and the demonstrator&#39;s actions&#39; effects on the environment. As a simple example according to further embodiments, assume one wants to use highway drone data, e.g. of the drone DR of  FIG. 6 , to learn how an acceleration action A of a “demonstrator car”  10  affects a lane changing behavior Z of a “lead car”  2  in front of it on a slower (e.g. right) lane L 1 . 
     According to further preferred embodiments, slightly simplifying the reality, assume the indicator light IL of the lead car  2  serves as a perfect coordination device: whenever it is on, it will result in (1) the demonstrator car  10  decelerating and (2) the lead car  2  changing lane to the fast lane L 2 . Now assume one just uses the variables recorded in the drone data of said drone(s) DR, where the indicator light IL is not contained, estimate P(ZIA) (probability of changing lane given the acceleration) from it. This may lead to the conclusion that an agent in the place of the demonstrator  10  can arbitrarily chose any acceleration or deceleration action, and the lead car  2  will perfectly adapt Z and only change lane when agent decelerates—which in practice can lead to crashes. 
     To at least partly mitigate these disadvantages, the principle according to the embodiments is proposed, as exemplarily depicted inter alia by  FIG. 2A  already discussed above. According to further preferred embodiments, said steps of determining  100  first data Dat_ 1  associated with a demonstrator  10  ( FIG. 1, 6 ) of said first domain D_ 1 , determining  110  first knowledge KN_ 1  from said first domain D_ 1  based on said first data Dat_ 1 , transferring  120  at least a part KN_ 1 ′ of said first knowledge KN_ 1  to a second domain D_ 2  enable learning from demonstrations (LfD) under sensor-shift, i.e., when a) the sensors of the demonstrator  10  and/or b) the sensors that are used to observe the demonstrator  10 , and/or c) the sensors of an AI agent that is to be trained depending on said demonstrator  10  are different. 
     According to further preferred embodiments, which are explained in detail further below, methods and techniques are proposed that enable to use causal models to, preferably rigorously, analyze, on a population-level, to what extent the relevant underlying mechanisms (the decision-effect and the demonstrator policy) can be identified and transferred from the available observations. Furthermore, further preferred embodiments propose algorithms to calculate them. 
     According to further preferred embodiments, proxy methods are introduced which may be easier to calculate, and/or to estimate from finite data and/or to interpret than the exact solutions, alongside theoretical bounds on their closeness to the exact ones. 
     Further preferred embodiments further comprise, cf.  FIG. 2B : modeling  130  the first (e.g., source) domain D_ 1  ( FIG. 1 ) by means of a directed acyclic graph, DAG, G 1  and/or modeling  132  the second (e.g., target) domain D_ 2  by means of a DAG G 2 . In this regard,  FIG. 3  schematically depicts the first DAG G 1  that models and/or characterizes said first domain D_ 1  and said second DAG G 2  that models and/or characterizes said second domain D_ 2 . 
     According to further preferred embodiments, the following variables may be used, e.g. within said DAGs G 1 , G 2 , cf.  FIG. 3 : variable “X” may characterize a state of the system, variable “A” may characterize an action of an agent, and variable “Z” may characterize (i.e., stand for) an outcome (e.g., an abstract variable that could e.g. be, say, the state of cars in a the next time instance given a state and an action). 
     According to further preferred embodiments, the following variables may be used, especially to characterize observations: the variable “Y D ” may characterize the demonstrator&#39;s input, e.g. generated by the demonstrator&#39;s sensor(s), the variable “Y S ” may characterize the AI agent&#39;s observation of the state of the source system (e.g., drone data from the high-way HW, in the context of the aforementioned example of  FIG. 6 ), and in the target domain, the variable “Y T ” may characterize an input to the target agent measured by its sensors. 
     According to further preferred embodiments, let the distributions over variables in the source domain and the target domain (e.g. P(Z)) be denoted by subscript “S” and “T”, respectively (e.g., P S (Z) and P T (Z)). According to further preferred embodiments, let π D (a|Y D ) denote a policy of the demonstrator  10  (also denoted as “expert”), and π T (a|Y T ) denote a policy of the target agent. 
     Further preferred embodiments relate to a method of designing a target agent that observes and successfully acts in the target domain D_ 2  ( FIG. 1 ), e.g., based on what is known and/or what may be derived from the source domain D_ 1  (e.g., from observing the demonstrator  10 ) and its relation to the target domain. 
     According to further preferred embodiments, said method further comprises, cf.  FIG. 2C : determining  140 , particularly inferring, an effect EF of an action to an outcome in the second domain D_ 2 , particularly conditional on an observation Y T  in said second domain D_ 2 . 
     According to further preferred embodiments, said method further comprises: providing  142  a utility function u(Z) associated with said outcome, and, optionally, determining  144  a first action al by maximizing said utility function u(Z) with respect to an action a, particularly based on said observation Y T , wherein said optional step of determining  144  may e.g. be performed based on the equation E[u(Z)|do(a), Y T ], wherein E[ ] is an expected value. 
     According to further preferred embodiments, said method further comprises, cf.  FIG. 2C : determining  150 , particularly inferring, a conditional distribution over actions a given an or said observation Y T  in the second domain D_ 2 , preferably such that a target agent  20  ( FIG. 1 ) associated with said second domain D_ 2  behaves similar to a or said demonstrator  10  ( FIG. 1 ) of the first domain. 
     According to further preferred embodiments, cf.  FIG. 7A , said method further comprises: a) using  160  the equation 
     
       
         
           
             
               
                 
                   p 
                   S 
                 
                  
                 
                   ( 
                   
                     z 
                     , 
                     a 
                     , 
                     
                       y 
                       S 
                     
                   
                   ) 
                 
               
               = 
               
                 
                   
                     ∞ 
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           y 
                           S 
                         
                          
                         x 
                       
                       ) 
                     
                   
                    
                   
                     ps 
                      
                     
                       ( 
                       
                         z 
                         , 
                         a 
                         , 
                         x 
                       
                       ) 
                     
                   
                 
                 = 
                 
                   
                     
                       x 
                       , 
                       
                         y 
                         D 
                       
                     
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           y 
                           S 
                         
                          
                         x 
                       
                       ) 
                     
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         
                           z 
                            
                           a 
                         
                         , 
                         x 
                       
                       ) 
                     
                   
                    
                   
                     
                       π 
                       D 
                     
                      
                     
                       ( 
                       
                         a 
                          
                         
                           y 
                           D 
                         
                       
                       ) 
                     
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           y 
                           D 
                         
                         , 
                         x 
                       
                       ) 
                     
                   
                 
               
             
             , 
           
         
       
     
     particularly characterizing one or more aspects of at least one of said DAGs G 1 , G 2  ( FIG. 3 ) with the equation 
     
       
         
           
             
               
                 
                   p 
                   S 
                 
                  
                 
                   ( 
                   
                     z 
                     , 
                     a 
                     , 
                     
                       y 
                       S 
                     
                   
                   ) 
                 
               
               = 
               
                 
                   
                     ∞ 
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           y 
                           S 
                         
                          
                         x 
                       
                       ) 
                     
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         z 
                         , 
                         a 
                         , 
                         x 
                       
                       ) 
                     
                   
                 
                 = 
                 
                   
                     
                       x 
                       , 
                       
                         y 
                         D 
                       
                     
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           y 
                           S 
                         
                          
                         x 
                       
                       ) 
                     
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         
                           z 
                            
                           a 
                         
                         , 
                         x 
                       
                       ) 
                     
                   
                    
                   
                     
                       π 
                       D 
                     
                      
                     
                       ( 
                       
                         a 
                          
                         
                           y 
                           D 
                         
                       
                       ) 
                     
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           y 
                           D 
                         
                         , 
                         x 
                       
                       ) 
                     
                   
                 
               
             
             , 
           
         
       
     
     and/or b) using  160 , particularly characterizing one or more aspects of at least one of said DAGs with, the equation 
     
       
         
           
             
               
                 
                   
                     
                       [ 
                       
                         
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   z 
                                   , 
                                   a 
                                   , 
                                   
                                     y 
                                     1 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                         
                           
                             ⋮ 
                           
                         
                         
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   z 
                                   , 
                                   a 
                                   , 
                                   
                                     y 
                                     m 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       ] 
                     
                     = 
                     
                       
                         [ 
                         
                           
                             
                               
                                 P 
                                  
                                 
                                   ( 
                                   
                                     
                                       y 
                                       1 
                                     
                                      
                                     
                                       x 
                                       1 
                                     
                                   
                                   ) 
                                 
                               
                             
                             
                               … 
                             
                             
                               
                                 P 
                                  
                                 
                                   ( 
                                   
                                     
                                       y 
                                       1 
                                     
                                      
                                     
                                       x 
                                        
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                           
                             
                               ⋮ 
                             
                             
                               
                                   
                               
                             
                             
                               ⋮ 
                             
                           
                           
                             
                               
                                 P 
                                  
                                 
                                   ( 
                                   
                                     
                                       y 
                                       m 
                                     
                                      
                                     
                                       x 
                                       1 
                                     
                                   
                                   ) 
                                 
                               
                             
                             
                               … 
                             
                             
                               
                                 P 
                                  
                                 
                                   ( 
                                   
                                     
                                       y 
                                       m 
                                     
                                      
                                     
                                       x 
                                        
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                         
                         ] 
                       
                        
                       
                         [ 
                         
                           
                             
                               
                                 P 
                                  
                                 
                                   ( 
                                   
                                     z 
                                     , 
                                     a 
                                     , 
                                     
                                       x 
                                       1 
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                           
                             
                               ⋮ 
                             
                           
                           
                             
                               
                                 P 
                                  
                                 
                                   ( 
                                   
                                     z 
                                     , 
                                     a 
                                     , 
                                     
                                       x 
                                        
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                         
                         ] 
                       
                     
                   
                   , 
                 
               
               
                 
                   [ 
                   
                     equation 
                      
                     
                         
                     
                      
                     1 
                   
                   ] 
                 
               
             
           
         
       
     
     preferably for all values of z and/or all values of a. 
     According to further preferred embodiments, said method further comprises: using  162 , particularly for said step of determining  140  ( FIG. 2C ), the equation 
     
       
         
           
             
               
                 
                   p 
                   ~ 
                 
                  
                 
                   ( 
                   
                     
                       
                         Z 
                          
                         X 
                       
                       = 
                       x 
                     
                     , 
                     A 
                   
                   ) 
                 
               
               := 
               
                 
                   ∑ 
                   y 
                 
                  
                 
                   
                     ps 
                      
                     
                       ( 
                       
                         
                           
                             Z 
                              
                             
                               Y 
                               S 
                             
                           
                           = 
                           y 
                         
                         , 
                         A 
                       
                       ) 
                     
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         
                           Y 
                           S 
                         
                         = 
                         
                           y 
                            
                           x 
                         
                       
                       ) 
                     
                   
                 
               
             
             , 
           
         
       
     
     wherein {tilde over (p)}(Z|X=x,A) characterizes a proxy for the conditional distribution of outcome given action and state (average-based action-effect proxy), wherein p S (Z|Y S =y,A) characterizes a conditional distribution of the outcome given the action and the observation in the source domain D_ 1 , and wherein p(Y S =y|x) characterizes a conditional distribution of the observation in the source domain D_ 1  given the state. 
     According to further preferred embodiments, said method further comprises: using  164  ( FIG. 7A ) at least one of the following equations, particularly for said step of determining  150  ( FIG. 2C ), particularly inferring, said conditional distribution over actions a: a) 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           
                             
                               
                                 
                                   π 
                                   ~ 
                                 
                                 T 
                                 
                                   ( 
                                   1 
                                   ) 
                                 
                               
                                
                               
                                 ( 
                                 
                                   
                                     a 
                                      
                                     
                                       Y 
                                       T 
                                     
                                   
                                   = 
                                   y 
                                 
                                 ) 
                               
                             
                             = 
                             
                               
                                 
                                   y 
                                   ′ 
                                 
                               
                                
                               
                                 
                                   p 
                                   S 
                                 
                                  
                                 
                                   ( 
                                   
                                     
                                       a 
                                        
                                       
                                         Y 
                                         S 
                                       
                                     
                                     = 
                                     
                                       y 
                                       ′ 
                                     
                                   
                                   ) 
                                 
                               
                                
                               
                                 
                                   p 
                                   S 
                                 
                                  
                                 
                                   ( 
                                   
                                     
                                       Y 
                                       S 
                                     
                                     = 
                                     
                                       
                                         
                                           y 
                                           ′ 
                                         
                                          
                                         
                                           Y 
                                           D 
                                         
                                       
                                       = 
                                       y 
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                           , 
                           b 
                         
                         ) 
                       
                        
                       
                         
 
                       
                        
                       
                         
                           
                             
                               
                                 π 
                                 ~ 
                               
                               T 
                               
                                 ( 
                                 2 
                                 ) 
                               
                             
                              
                             
                               ( 
                               
                                 a 
                                  
                                 
                                   y 
                                   T 
                                 
                               
                               ) 
                             
                           
                           ∝ 
                           
                             exp 
                              
                             
                               ( 
                               
                                 
                                   
                                     y 
                                     S 
                                   
                                 
                                  
                                 
                                   p 
                                    
                                   
                                     ( 
                                     
                                       
                                         y 
                                         S 
                                       
                                        
                                       
                                         y 
                                         T 
                                       
                                     
                                     ) 
                                   
                                 
                                  
                                 log 
                                  
                                 
                                     
                                 
                                  
                                 
                                   
                                     p 
                                     S 
                                   
                                    
                                   
                                     ( 
                                     
                                       a 
                                        
                                       
                                         y 
                                         S 
                                       
                                     
                                     ) 
                                   
                                 
                               
                               ) 
                             
                           
                         
                         , 
                         c 
                       
                     
                     ) 
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         
                           π 
                           ~ 
                         
                         T 
                         
                           ( 
                           3 
                           ) 
                         
                       
                        
                       
                         ( 
                         
                           a 
                            
                           
                             y 
                             T 
                           
                         
                         ) 
                       
                     
                     ∝ 
                     
                       
                         exp 
                          
                         
                           ( 
                           
                             
                               x 
                             
                              
                             
                               p 
                                
                               
                                 ( 
                                 
                                   x 
                                    
                                   
                                     y 
                                     T 
                                   
                                 
                                 ) 
                               
                             
                              
                             log 
                              
                             
                               
                                 p 
                                 ~ 
                               
                                
                               
                                 ( 
                                 
                                   a 
                                    
                                   x 
                                 
                                 ) 
                               
                             
                           
                           ) 
                         
                       
                       . 
                     
                   
                 
               
               
                 
                     
                 
               
             
           
         
       
     
     According to further preferred embodiments, said step of using  164  further comprises, cf.  FIG. 7B : a) using  164   a , if a or said observation in said second domain D_ 2  is the same as an expert&#39;s observation, the equation 
     
       
         
           
             
               
                 
                   
                     π 
                     ~ 
                   
                   T 
                   
                     ( 
                     1 
                     ) 
                   
                 
                  
                 
                   ( 
                   
                     
                       a 
                        
                       
                         Y 
                         T 
                       
                     
                     = 
                     y 
                   
                   ) 
                 
               
               = 
               
                 
                   
                     y 
                     ′ 
                   
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         a 
                          
                         
                           Y 
                           S 
                         
                       
                       = 
                       
                         y 
                         ′ 
                       
                     
                     ) 
                   
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         Y 
                         S 
                       
                       = 
                       
                         
                           
                             y 
                             ′ 
                           
                            
                           
                             Y 
                             D 
                           
                         
                         = 
                         y 
                       
                     
                     ) 
                   
                 
               
             
             , 
           
         
       
     
     and/or b) using  164   b , if demonstrator&#39;s sensors and a target agent&#39;s sensors in the first domain D_ 1  are the same, but if the target agent&#39;s sensors in the first domain D_ 1  are different from the target agent&#39;s sensors in the second domain D_ 2 , the equation 
     
       
         
           
             
               
                 
                   π 
                   ~ 
                 
                 T 
                 
                   ( 
                   2 
                   ) 
                 
               
                
               
                 ( 
                 
                   a 
                    
                   
                     y 
                     T 
                   
                 
                 ) 
               
             
             ∝ 
             
               exp 
                
               
                 ( 
                 
                   
                     
                       y 
                       S 
                     
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         
                           y 
                           S 
                         
                          
                         
                           y 
                           T 
                         
                       
                       ) 
                     
                   
                    
                   log 
                    
                   
                       
                   
                    
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         a 
                          
                         
                           y 
                           S 
                         
                       
                       ) 
                     
                   
                 
                 ) 
               
             
           
         
       
     
     and/or c) using  164   c , if the demonstrator&#39;s sensors and the target agent&#39;s sensors in the first domain D_ 1  and the target agent&#39;s sensors in the second domain D_ 2  are each different from each other, the equation 
     
       
         
           
             
               
                 
                   
                     π 
                     ~ 
                   
                   T 
                   
                     ( 
                     3 
                     ) 
                   
                 
                  
                 
                   ( 
                   
                     a 
                      
                     
                       y 
                       T 
                     
                   
                   ) 
                 
               
               ∝ 
               
                 exp 
                  
                 
                   ( 
                   
                      
                     
                       p 
                        
                       
                         ( 
                         
                           x 
                            
                           
                             y 
                             T 
                           
                         
                         ) 
                       
                     
                      
                     log 
                      
                     
                       
                         p 
                         ~ 
                       
                        
                       
                         ( 
                         
                           a 
                            
                           x 
                         
                         ) 
                       
                     
                   
                   ) 
                 
               
             
             , 
             
               
                 wherein 
                  
                 
                     
                 
                  
                 
                   
                     p 
                     ~ 
                   
                    
                   
                     ( 
                     
                       a 
                        
                       x 
                     
                     ) 
                   
                 
               
               := 
               
                 
                   y 
                 
                  
                 
                   
                     p 
                     S 
                   
                    
                   
                     ( 
                     
                       
                         a 
                          
                         
                           Y 
                           S 
                         
                       
                       = 
                       y 
                     
                     ) 
                   
                 
                  
                 
                   
                     
                       p 
                       S 
                     
                      
                     
                       ( 
                       
                         
                           Y 
                           S 
                         
                         = 
                         
                           y 
                            
                           x 
                         
                       
                       ) 
                     
                   
                   . 
                 
               
             
           
         
       
     
     Further preferred embodiments relate to an apparatus  200 , cf.  FIG. 4 , configured to perform the method according to the embodiments. The apparatus  200  comprises at least one calculating unit  202  (“computer”) and at least one memory unit  204  associated with (i.e., usably by) said at least one calculating unit  202  for at least temporarily storing a computer program PRG and/or data DAT, wherein said computer program PRG is, e.g., configured to at least temporarily control an operation of said apparatus  200 , e.g. the execution of a method according to the embodiments, for example for performing an LfD-technique based on e.g. the method according to  FIG. 2A . 
     According to further preferred embodiments, said at least one calculating unit  202  is configured to execute said computer program PRG or at least parts thereof, e.g. for executing the method according to the embodiments or at least one or more steps thereof. 
     According to further preferred embodiments, said at least one calculating unit  202  may comprise at least one of the following elements: a microprocessor, a microcontroller, a digital signal processor (DSP), a programmable logic element (e.g., FPGA, field programmable gate array), an ASIC (application specific integrated circuit), hardware circuitry, a graphics processor (GPU), a tensor processor. According to further preferred embodiments, any combination of two or more of these elements is also possible. 
     According to further preferred embodiments, the memory unit  204  comprises at least one of the following elements: a volatile memory  204   a , particularly a random-access memory (RAM), a non-volatile memory  204   b , particularly a Flash-EEPROM. Preferably, said computer program PRG is at least temporarily stored in said non-volatile memory  204   b . Data DAT, which may e.g. be used for executing the method according to the embodiments, may at least temporarily be stored in said RAM  204   a . Said data DAT may e.g. comprise said first data Dat_ 1  ( FIG. 2A ), i.e. sensor data or data derived from said sensor data, e.g. characterizing and/or representing observations Y S  and/or Y D  and/or Y T  as explained above. 
     According to further preferred embodiments, an optional computer-readable storage medium SM comprising instructions, e.g. in the form of a further computer program PRG′, may be provided, wherein said further computer program PRG′, when executed by a computer, i.e. by the calculating unit  202 , may cause the computer  202  to carry out the method according to the embodiments. As an example, said storage medium SM may comprise or represent a digital storage medium such as a semiconductor memory device (e.g., solid state drive, SSD) and/or a magnetic storage medium such as a disk or harddisk drive (HDD) and/or an optical storage medium such as a compact disc (CD) or DVD (digital versatile disc) or the like. 
     According to further preferred embodiments, the apparatus  200  may comprise an optional data interface  206 , preferably for bidirectional data exchange with an external device (not shown). As an example, by means of said data interface  206 , a data carrier signal DCS may be received, e.g. from said external device, for example via a wired or a wireless data transmission medium, e.g. over a (virtual) private computer network and/or a public computer network such as e.g. the Internet. According to further preferred embodiments, the data carrier signal DCS may represent or carry the computer program PRG according to the embodiments, or at least a part thereof. 
     According to further preferred embodiments, the apparatus  200  may also be used to provide a machine learning system MLS, e.g. performing one or more LfD techniques, preferably incorporating a method according to preferred embodiments, cf. e.g.  FIG. 2A to 2C . 
       FIG. 5A  schematically depicts a simplified block diagram according to further preferred embodiments, wherein said block diagram of  FIG. 5A  may e.g. be associated with a first task exemplarily characterized by at least one of the steps  140  ( FIG. 2C ),  164  ( FIG. 7A ) explained above according to further preferred embodiments. Block B 1  of  FIG. 5A  characterizes a source domain, i.e. the first domain D_ 1  ( FIG. 1 ), block B 2  characterizes a target domain, i.e. the second domain D_ 2  ( FIG. 1 ), block B 3  characterizes knowledge, e.g. said first knowledge KN_ 1  ( FIG. 2A ) or at least a part KN_ 1 ′ thereof, block B 4  characterizes an outcome Z and/or observations Y S , block B 5  characterizes an outcome Z and/or observations Y T , block B 6  characterizes an effect of action A on the outcome Z given the observations Y T , i.e. according to p(Z|do(A), Y T ), and block B 7  characterizes a selection of an optimal action A with respect to the outcome Z given the observations Y T . 
       FIG. 5B  schematically depicts a simplified block diagram according to further preferred embodiments, wherein said block diagram of  FIG. 5B  may e.g. be associated with a second task exemplarily characterized by at least one of the steps  164  ( FIG. 7A ),  164   a ,  164   b ,  164   c  ( FIG. 7B ) explained above according to further preferred embodiments. 
     Similar to  FIG. 5A , block B 1  of  FIG. 5B  characterizes the source domain, i.e. the first domain D_ 1  ( FIG. 1 ), block B 2  characterizes the target domain, i.e. the second domain D_ 2  ( FIG. 1 ), and block B 3  characterizes knowledge, e.g. said first knowledge KN_ 1  ( FIG. 2A ) or at least a part KN_ 1 ′ thereof. Block B 4 ′ of  FIG. 5B  characterizes observations Y S , block B 5  characterizes observations Y T , and block B 6 ′ characterizes a policy for an imitator, i.e. the target agent  20  ( FIG. 1 ), wherein said policy may e.g. be characterized by Π(A|Y T ). 
     Further preferred embodiments relate to a use 170, cf.  FIG. 8 , of the method according to the embodiments and/or of the apparatus  200  ( FIG. 4 ) according to the embodiments and/or of the computer program PRG, PRG′ according to the embodiments for at least one of: a) training  170   a  ( FIG. 8 ) a machine learning system MLS ( FIG. 2 ), b) determining  170   b  and/or providing training data TD, particularly for a machine learning system MLS, c) learning  170   c  from demonstrations, particularly under sensor-shift, i.e., when the sensors of the demonstrator  10 , the sensors that are used to observe the demonstrator  10 , and the sensors of the AI agent that is to be trained are different, d) controlling  170   d  movement of a physical system such as a robot or an autonomous car. 
     According to further preferred embodiments, the principle according to the embodiments can be used to receive sensor signals (e.g., as said first data Dat_ 1 , cf.  FIG. 2A ), e.g. from at least one sensor such as, e.g., video, radar, LiDAR, ultrasonic, motion, and the like, to name a few, wherein, e.g., at least parts of said sensor signals may be processed as said data associated with the first domain D_ 1  according to preferred embodiments. 
     According to further preferred embodiments, the principle according to the embodiments can be used to compute one or more control signals for controlling a physical system, like e.g. a computer-controlled machine, like a robot, a vehicle, a manufacturing machine, a personal assistant or an access control system or the like. Advantageously, LfD-techniques may be used according to further preferred embodiments, even if there is a non-vanishing sensor-shift. 
     Further preferred embodiments may comprise at least one of the following steps: a) analyzing and/or processing the sensor data, b) incorporating specifications of sensors in both the source domain (first domain) D_ 1  and the target (second) domain(s) D_ 2 , c) transfer extracted knowledge from the source domain D_ 1  to the target domain D_ 2 . 
     According to further preferred embodiments, the principle according to the embodiments can be used to train a machine learning system MLS ( FIG. 4 ) that can be used for e.g. the above applications. 
     According to further preferred embodiments, the principle according to the embodiments can be used to generate training data TD (cf. step  170   b  of  FIG. 8 ) for training a machine learning system MLS that can e.g. be used for the above applications. 
     In the following paragraphs, further preferred embodiments and exemplary aspects are exemplarily disclosed with reference to an algorithm as presented in table 1 below. Table 1 has two columns, wherein a first column identifies a line number, and wherein a second column comprises pseudocode representing elements of said algorithm. The algorithm of table 1 may, according to further preferred embodiments, e.g. be used to parameterize a solution set of the matrix equation [equation 1] mentioned above, e.g. with respect to step  160  of  FIG. 7A . 
     According to further preferred embodiments, inputs to the algorithm of table 1 are: P(z, a, Y S ), i.e. the left hand side of [equation 1] and [P(y i |x j )] i=1,j=1   m,l , i.e. the matrix of [equation 1]. 
     According to further preferred embodiments, the output of the algorithm of table 1 is: ζ 1 , . . . , ζkϵ , such that their convex hull is the solution set to (3) 
     
       
         
           
               
             
               
                   
               
             
            
               
                 *** Start of Table 1: algorithm 1 *** 
               
            
           
           
               
               
            
               
                 1 
                 Rearrange columns of [P(y i |x j )] i,j=1   m,l  such that 
               
            
           
           
               
               
               
            
               
                   
                   
                 [P(y i |x j )] i,j=1   m,l  = [D E] amd D∈   is non-singular: 
               
            
           
           
               
               
            
               
                 2 
                 UΣV T  ← SVD of [P(y i |x j )] i,j=1   m,l ; 
               
               
                 3 
                 for i = 1 to l − m do 
               
            
           
           
               
               
               
               
            
               
                 4 
                   
                 └ 
                     ← zero vector of length l − m whose ith entry is one: 
               
               
                   
               
            
           
           
               
               
            
               
                 5 
                 
                   
                     
                       
                         
                           M 
                           ← 
                           
                             V 
                              
                             
                               [ 
                               
                                 
                                   
                                     0 
                                   
                                   
                                     ⋯ 
                                   
                                   
                                     0 
                                   
                                 
                                 
                                   
                                     
                                       ? 
                                     
                                   
                                   
                                     ⋯ 
                                   
                                   
                                     
                                       ? 
                                     
                                   
                                 
                               
                               ] 
                             
                           
                         
                         , 
                         
                             
                         
                          
                         
                           
                             b 
                             ← 
                             
                               [ 
                               
                                 
                                   
                                     
                                       
                                         D 
                                         
                                           - 
                                           1 
                                         
                                       
                                        
                                       
                                         P 
                                          
                                         
                                           ( 
                                           
                                             z 
                                             , 
                                             a 
                                             , 
                                             Ys 
                                           
                                           ) 
                                         
                                       
                                     
                                   
                                 
                                 
                                   
                                     0 
                                   
                                 
                               
                               ] 
                             
                           
                           ; 
                         
                       
                     
                   
                 
               
               
                   
               
               
                 6 
                 i ←1; 
               
               
                 7 
                 for any sub-matrix R of M with dimension 
               
            
           
           
               
               
               
            
               
                   
                   
                 (l − m) × (l − m) do 
               
            
           
           
               
               
               
               
            
               
                 8 
                   
                 | 
                 {circumflex over (b)} ← the sub-vector of b of length l − m that 
               
               
                   
                   
                 | 
                 corresponds to the selected rows of M; 
               
               
                 9 
                   
                 | 
                 if R −1  exists and −M R −1 b + b ≥ 0 then 
               
               
                 10 
                   
                 |  
                 | ζ    ← −M R −1 b + b; 
               
               
                 11 
                   
                 └ 
                 └ i ← i + 1; 
               
            
           
           
               
            
               
                 *** End of Table 1: algorithm 1 *** 
               
               
                   
               
               
                     indicates data missing or illegible when filed 
               
            
           
         
       
     
     According to further preferred, exemplary and particularly non-limiting, embodiments, in the following, each line of algorithm  1  is explained in further detail. 
     Line 1: rearrange the columns of matrix [P(y i |x j )] i=1,j=1   m,l  to obtain a matrix which, for simplicity, we denote again by [P(y i |x j )] i=1,j=1   m,l , such that the first m columns of it (denoted by D) becomes a non-singular matrix (the remaining columns are denoted by E). 
     Line 2: After rearrangement, singular value decomposition (SVD) is been applied to obtain matrices U, V, and Σ. SVD is a factorization method that finds a factorization of a given matrix say [P(y i |x j )] i=1,j=1   m,l  based on its singular values. 
     Lines 3 and 4: construct a set of l−m unity vectors e i  in which all entries are zero except one entry. 
     Line 5: construct matrix M and vector b using the previously constructed vectors and matrices. 
     Line 6: start a counter by setting in to 1. 
     Line 7 to 11: using a for-loop obtain the outputs. In this loop the following occurs 
     Line 8: construct a sub-vector {circumflex over (b)} of b with length l−m that are selected according to the selection of sub-matrix R. 
     Line 9: check if the selected sub-matrix R is non-singular (has inverse) and further if −MR −1 {circumflex over (b)}+b is a positive vector. 
     Line 10 and 11: if the conditions are satisfied the output vector ζ i  will be constructed and the counter will be increased by one.