Title: ZERO-SHOT VISUAL IMITATION

Abstract: The current dominant paradigm for imitation learning relies on strong supervision of expert actions to learn both what and how to imitate. We pursue an alternative paradigm wherein an agent first explores the world without any expert supervision and then distills its experience into a goal-conditioned skill policy with a novel forward consistency loss. In our framework, the role of the expert is only to communicate the goals (i.e., what to imitate) during inference. The learned policy is then employed to mimic the expert (i.e., how to imitate) after seeing just a sequence of images demonstrating the desired task. Our method is "zero-shot" in the sense that the agent never has access to expert actions during training or for the task demonstration at inference. We evaluate our zero-shot imitator in two real-world settings: complex rope manipulation with a Baxter robot and navigation in previously unseen office environments with a TurtleBot. Through further experiments in VizDoom simulation, we provide evidence that better mechanisms for exploration lead to learning a more capable policy which in turn improves end task performance. Videos, models, and more details are available at https://pathak22.github.io/zeroshot-imitation/.

1. INTRODUCTION:

Imitating expert demonstration is a powerful mechanism for learning to perform tasks from raw sensory observations. The current dominant paradigm in learning from demonstration (LfD) (Argall et al., 2009; Ng & Russell, 2000; Pomerleau, 1989; Schaal, 1999) requires the expert to either manually move the robot joints (i.e., kinesthetic teaching) or teleoperate the robot to execute the desired task. The expert typically provides multiple demonstrations of a task at training time, and this generates data in the form of observation-action pairs from the agent's point of view. The agent then distills this data into a policy for performing the task of interest. Such a heavily supervised approach, where it is necessary to provide demonstrations by controlling the robot, is incredibly tedious for the human expert. Moreover, for every new task that the robot needs to execute, the expert is required to provide a new set of demonstrations.

Instead of communicating how to perform a task via observation-action pairs, a more general formulation allows the expert to communicate only what needs to be done by providing the observations of the desired world states via a video or a sparse sequence of images. This way, the agent is required to infer how to perform the task (i.e., actions) by itself. In psychology, this is known as observational learning (Bandura & Walters, 1977) . While this is a harder learning problem, it is a more interesting setting, because the expert can demonstrate multiple tasks quickly and easily.

An agent without any prior knowledge will find it extremely hard to imitate a task by simply watching a visual demonstration in all but the simplest of cases. Thus, the natural question is: in order to imitate, what form of prior knowledge must the agent possess? A large body of work (Breazeal & Scassellati, 2002; Dillmann, 2004; Ikeuchi & Suehiro, 1994; Kuniyoshi et al., 1989; 1994; Yang et al., 2015) has sought to capture prior knowledge by manually pre-defining the state that must be feed-forward < l a t e x i t s h a 1 _ b a s e 6 4 = " A W l 8 5 E u k k 0 w 0 X t y q h X M H P z 8 x 0 K s = " > A A A B 8 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B i y X p R b 0 V v H i s Y G y h D W W z m b R L N 5 u w u 1 F K 6 d / w 4 k H F q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B N c G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f D o Q a e 5 Y u i z V K S q E 1 K N g k v 0 D T c C O 5 l C m o Q C 2 + H o Z u a 3 H 1 F p n s p 7 M 8 4 w S O h A 8 p g z a q z U i x G j i z h V T 1 R F / W r N r b t z k F X i F a Q G B V r 9 6 l c v S l m e o D R M U K 2 7 n p u Z Y E K V 4 U z g t N L L N W a U j e g A u 5 Z K m q A O J v O b p + T M K h G x q 2 1 J Q + b q 7 4 k J T b Q e J 6 H t T K g Z 6 m V v J v 7 n d X M T X w U T L r P c o G S L R X E u i E n J L A A S c Y X M i L E l l C l u b y V s S B V l x s Z U s S F 4 y y + v E r 9 R v 6 5 7 d 4 1 a s 1 G k U Y Y T O I V z 8 O A S m n A L L f C B Q Q b P 8 A p v T u 6 8 O O / O x 6 K 1 5 B Q z x / A H z u c P a L y R a g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A W l 8 5 E u k k 0 w 0 X t y q h X M H P z 8 x 0 K s = " > A A A B 8 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B i y X p R b 0 V v H i s Y G y h D W W z m b R L N 5 u w u 1 F K 6 d / w 4 k H F q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B N c G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f D o Q a e 5 Y u i z V K S q E 1 K N g k v 0 D T c C O 5 l C m o Q C 2 + H o Z u a 3 H 1 F p n s p 7 M 8 4 w S O h A 8 p g z a q z U i x G j i z h V T 1 R F / W r N r b t z k F X i F a Q G B V r 9 6 l c v S l m e o D R M U K 2 7 n p u Z Y E K V 4 U z g t N L L N W a U j e g A u 5 Z K m q A O J v O b p + T M K h G x q 2 1 J Q + b q 7 4 k J T b Q e J 6 H t T K g Z 6 m V v J v 7 n d X M T X w U T L r P c o G S L R X E u i E n J L A A S c Y X M i L E l l C l u b y V s S B V l x s Z U s S F 4 y y + v E r 9 R v 6 5 7 d 4 1 a s 1 G k U Y Y T O I V z 8 O A S m n A L L f C B Q Q b P 8 A p v T u 6 8 O O / O x 6 K 1 5 B Q z x / A H z u c P a L y R a g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A W l 8 5 E u k k 0 w 0 X t y q h X M H P z 8 x 0 K s = " > A A A B 8 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B i y X p R b 0 V v H i s Y G y h D W W z m b R L N 5 u w u 1 F K 6 d / w 4 k H F q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B N c G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f D o Q a e 5 Y u i z V K S q E 1 K N g k v 0 D T c C O 5 l C m o Q C 2 + H o Z u a 3 H 1 F p n s p 7 M 8 4 w S O h A 8 p g z a q z U i x G j i z h V T 1 R F / W r N r b t z k F X i F a Q G B V r 9 6 l c v S l m e o D R M U K 2 7 n p u Z Y E K V 4 U z g t N L L N W a U j e g A u 5 Z K m q A O J v O b p + T M K h G x q 2 1 J Q + b q 7 4 k J T b Q e J 6 H t T K g Z 6 m V v J v 7 n d X M T X w U T L r P c o G S L R X E u i E n J L A A S c Y X M i L E l l C l u b y V s S B V l x s Z U s S F 4 y y + v E r 9 R v 6 5 7 d 4 1 a s 1 G k U Y Y T O I V z 8 O A S m n A L L f C B Q Q b P 8 A p v T u 6 8 O O / O x 6 K 1 5 B Q z x / A H z u c P a L y R a g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A W l 8 5 E u k k 0 w 0 X t y q h X M H P z 8 x 0 K s = " > A A A B 8 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B i y X p R b 0 V v H i s Y G y h D W W z m b R L N 5 u w u 1 F K 6 d / w 4 k H F q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B N c G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f D o Q a e 5 Y u i z V K S q E 1 K N g k v 0 D T c C O 5 l C m o Q C 2 + H o Z u a 3 H 1 F p n s p 7 M 8 4 w S O h A 8 p g z a q z U i x G j i z h V T 1 R F / W r N r b t z k F X i F a Q G B V r 9 6 l c v S l m e o D R M U K 2 7 n p u Z Y E K V 4 U z g t N L L N W a U j e g A u 5 Z K m q A O J v O b p + T M K h G x q 2 1 J Q + b q 7 4 k J T b Q e J 6 H t T K g Z 6 m V v J v 7 n d X M T X w U T L r P c o G S L R X E u i E n J L A A S c Y X M i L E l l C l u b y V s S B V l x s Z U s S F 4 y y + v E r 9 R v 6 5 

f m 3 9 W s t U U a h S y a W q h U Q D Z w K q h h k O t U Q B i U M O N 2 H 3 Z K D f 3 I L S T I o r 0 0 u g E Z O 2 Y C 1 G i b F U M 3 8 U h N B m I q M g D K i + W 4 x K + F S q O 6 K i H S q F t h d Y B Z 9 d V o L A L d q t u u Q G I K J v Q z N f 8 M r e s P A k 8 M e g g M Z V a e a f g 0 j S N L Z 2 y o n W d d 9 L T C M j y j D K o e 8 G q Y a E 0 C 5 p Q 9 1 C Q W L Q j W z 4 a B 9 v W y b C L a l s 2 6 u G 7 E 9 H R m K t e 3 F o J 2 N i O v q v N i D / 0 + q p a R 0 0 M i a S 1 L 5 L R 4 t a K c d G 4 k F q O G I K q O E 9 C w h V z N 6 K a Y c o Q m 0 G 2 r U h + H 9 f n g T V 3 f J h 2 b / Y L R z v j d P I o U 2 0 h Y r I R / v o G J 2 j C q o i i u 7 R I 3 p B r 8 6 D 8 + S 8 O e + j 0 S l n 7 N l A v 8 r 5 / A K n c a X m < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A v M 0 k 9 A 7 Q n q k f A C e B S / 9 I O K X x 8 o = " > A A A C J 3 i c b V B N S w M x E M 3 6 W d e v q k c v w S K 0 B 8 u u C O p J Q V C P F a 0 W u q V k s 9 M 2 N J s s S V Y p S 3 + O F / + K F x E V P f p P T D 8 E t Q 4 M P N 6 b x 8 y 8 M O F M G 8 / 7 c K a m Z 2 b n 5 n M L 7 u L S 8 s p q f m 3 9 W s t U U a h S y a W q h U Q D Z w K q h h k O t U Q B i U M O N 2 H 3 Z K D f 3 I L S T I o r 0 0 u g E Z O 2 Y C 1 G i b F U M 3 8 U h N B m I q M g D K i + W 4 x K + F S q O 6 K i H S q F t h d Y B Z 9 d V o L A L d q t u u Q G I K J v Q z N f 8 M r e s P A k 8 M e g g M Z V a e a f g 0 j S N L Z 2 y o n W d d 9 L T C M j y j D K o e 8 G q Y a E 0 C 5 p Q 9 1 C Q W L Q j W z 4 a B 9 v W y b C L a l s 2 6 u G 7 E 9 H R m K t e 3 F o J 2 N i O v q v N i D / 0 + q p a R 0 0 M i a S 1 L 5 L R 4 t a K c d G 4 k F q O G I K q O E 9 C w h V z N 6 K a Y c o Q m 0 G 2 r U h + H 9 f n g T V 3 f J h 2 b / Y L R z v j d P I o U 2 0 h Y r I R / v o G J 2 j C q o i i u 7 R I 3 p B r 8 6 D 8 + S 8 O e + j 0 S l n 7 N l A v 8 r 5 / A K n c a X m < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A v M 0 k 9 A 7 Q n q k f A C e B S / 9 I O K X x 8 o = " > A A A C J 3 i c b V B N S w M x E M 3 6 W d e v q k c v w S K 0 B 8 u u C O p J Q V C P F a 0 W u q V k s 9 M 2 N J s s S V Y p S 3 + O F / + K F x E V P f p P T D 8 E t Q 4 M P N 6 b x 8 y 8 M O F M G 8 / 7 c K a m Z 2 b n 5 n M L 7 u L S 8 s p q f m 3 9 W s t U U a h S y a W q h U Q D Z w K q h h k O t U Q B i U M O N 2 H 3 Z K D f 3 I L S T I o r 0 0 u g E Z O 2 Y C 1 G i b F U M 3 8 U h N B m I q M g D K i + W 4 x K + F S q O 6 K i H S q F t h d Y B Z 9 d V o L A L d q t u u Q G I K J v Q z N f 8 M r e s P A k 8 M e g g M Z V a e a f g 0 j S N L Z 2 y o n W d d 9 L T C M j y j D K o e 8 G q Y a E 0 C 5 p Q 9 1 C Q W L Q j W z 4 a B 9 v W y b C L a l s 2 6 u G 7 E 9 H R m K t e 3 F o J 2 N i O v q v N i D / 0 + q p a R 0 0 M i a S 1 L 5 L R 4 t a K c d G 4 k F q O G I K q O E 9 C w h V z N 6 K a Y c o Q m 0 G 2 r U h + H 9 f n g T V 3 f J h 2 b / Y L R z v j d P I o U 2 0 h Y r I R / v o G J 2 j C q o i i u 7 R I 3 p B r 8 6 D 8 + S 8 O e + j 0 S l n 7 N l A v 8 r 5 / A K n c a X m < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A v M 0 k 9 A 7 Q n q k f A C e B S / 9 I O K X x 8 o = " > A A A C J 3 i c b V B N S w M x E M 3 6 W d e v q k c v w S K 0 B 8 u u C O p J Q V C P F a 0 W u q V k s 9 M 2 N J s s S V Y p S 3 + O F / + K F x E V P f p P T D 8 E t Q 4 M P N 6 b x 8 y 8 M O F M G 8 / 7 c K a m Z 2 b n 5 n M L 7 u L S 8 s p q f m 3 9 W s t U U a h S y a W q h U Q D Z w K q h h k O t U Q B i U M O N 2 H 3 Z K D f 3 I L S T I o r 0 0 u g E Z O 2 Y C 1 G i b F U M 3 8 U h N B m I q M g D K i + W 4 x K + F S q O 6 K i H S q F t h d Y B Z 9 d V o L A L d q t u u Q G I K J v Q z N f 8 M r e s P A k 8 M e g g M Z V a e a f g 0 j S N L Z 2 y o n W d d 9 L T C M j y j D K o e 8 G q Y a E 0 C 5 p Q 9 1 C Q W L Q j W z 4 a B 9 v W y b C L a l s 2 6 u G 7 E 9 H R m K t e 3 F o J 2 N i O v q v N i D / 0 + q p a R 0 0 M i a S 1 L 5 L R 4 t a K c d G 4 k F q O G I K q O E 9 C w h V z N 6 K a Y c o Q m 0 G 2 r U h + H 9 f n g T V 3 f J h 2 b / Y L R z v j d P I o U 2 0 h Y r I R / v o G J 2 j C q o i i u 7 R I 3 p B r 8 6 D 8 + S 8 O e + j 0 S l n 7 N l A v 8 r 5 / A K n c a X m < / l a t e x i t > (c) Forward-regularized GSP < l a t e x i t s h a 1 _ b a s e 6 4 = " Q s s F B y k 0 c u c C C Z p h g 4 F x 0 + O m J U k = " > A A A C H 3 i c b V D L S g M x F M 3 4 r O N r 1 K W b Y B H q w j J T h O q u I K j L i t Y W 2 l I y m d s 2 N J M Z k o x S h 3 6 K G 3 / F j Q s V c d e / M X 0 I 2 n o g c D j n X G 7 u 8 W P O l H b d o b W w u L S 8 s p p Z s 9 c 3 N r e 2 n Z 3 d O x U l k k K F R j y S N Z 8 o 4 E x A R T P N o R Z L I K H P o e r 3 z k d + 9 R 6 k Y p G 4 1 f 0 Y m i H p C N Z m l G g j t Z x i w 4 c O E y k F o U E O 7 B w 9 w h e R f C A y O J b Q S T i R 7 B E C f H l T t h s g g p 9 g y 8 m 6 e X c M P E + 8 K c m i K c o t 5 6 s R R D Q J z T j l R K m 6 5 8 a 6 m R K p G e U w s B u J g p j Q H u l A 3 V B B Q l D N d H z g A B 8 a J c D t S J o n N B 6 r v y d S E i r V D 3 2 T D I n u q l l v J P 7 n 1 R P d P m 2 m T M S J B k E n i 9 o J x z r C o 7 Z w w C R Q z f u G E C q Z + S u m X S I J N R 0 o 2 5 T g z Z 4 8 T y q F / F n e u y 5 k S y f T N j J o H x 2 g H P J Q E Z X Q F S q j C q L o C b 2 g N / R u P V u v 1 o f 1 O Y k u W N O Z P f Q H 1 v A b A i C j D Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Q s s F B y k 0 c u c C C Z p h g 4 F x 0 + O m J U k = " > A A A C H 3 i c b V D L S g M x F M 3 4 r O N r 1 K W b Y B H q w j J T h O q u I K j L i t Y W 2 l I y m d s 2 N J M Z k o x S h 3 6 K G 3 / F j Q s V c d e / M X 0 I 2 n o g c D j n X G 7 u 8 W P O l H b d o b W w u L S 8 s p p Z s 9 c 3 N r e 2 n Z 3 d O x U l k k K F R j y S N Z 8 o 4 E x A R T P N o R Z L I K H P o e r 3 z k d + 9 R 6 k Y p G 4 1 f 0 Y m i H p C N Z m l G g j t Z x i w 4 c O E y k F o U E O 7 B w 9 w h e R f C A y O J b Q S T i R 7 B E C f H l T t h s g g p 9 g y 8 m 6 e X c M P E + 8 K c m i K c o t 5 6 s R R D Q J z T j l R K m 6 5 8 a 6 m R K p G e U w s B u J g p j Q H u l A 3 V B B Q l D N d H z g A B 8 a J c D t S J o n N B 6 r v y d S E i r V D 3 2 T D I n u q l l v J P 7 n 1 R P d P m 2 m T M S J B k E n i 9 o J x z r C o 7 Z w w C R Q z f u G E C q Z + S u m X S I J N R 0 o 2 5 T g z Z 4 8 T y q F / F n e u y 5 k S y f T N j J o H x 2 g H P J Q E Z X Q F S q j C q L o C b 2 g N / R u P V u v 1 o f 1 O Y k u W N O Z P f Q H 1 v A b A i C j D Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Q s s F B y k 0 c u c C C Z p h g 4 F x 0 + O m J U k = " > A A A C H 3 i c b V D L S g M x F M 3 4 r O N r 1 K W b Y B H q w j J T h O q u I K j L i t Y W 2 l I y m d s 2 N J M Z k o x S h 3 6 K G 3 / F j Q s V c d e / M X 0 I 2 n o g c D j n X G 7 u 8 W P O l H b d o b W w u L S 8 s p p Z s 9 c 3 N r e 2 n Z 3 d O x U l k k K F R j y S N Z 8 o 4 E x A R T P N o R Z L I K H P o e r 3 z k d + 9 R 6 k Y p G 4 1 f 0 Y m i H p C N Z m l G g j t Z x i w 4 c O E y k F o U E O 7 B w 9 w h e R f C A y O J b Q S T i R 7 B E C f H l T t h s g g p 9 g y 8 m 6 e X c M P E + 8 K c m i K c o t 5 6 s R R D Q J z T j l R K m 6 5 8 a 6 m R K p G e U w s B u J g p j Q H u l A 3 V B B Q l D N d H z g A B 8 a J c D t S J o n N B 6 r v y d S E i r V D 3 2 T D I n u q l l v J P 7 n 1 R P d P m 2 m T M S J B k E n i 9 o J x z r C o 7 Z w w C R Q z f u G E C q Z + S u m X S I J N R 0 o 2 5 T g z Z 4 8 T y q F / F n e u y 5 k S y f T N j J o H x 2 g H P J Q E Z X Q F S q j C q L o C b 2 g N / R u P V u v 1 o f 1 O Y k u W N O Z P f Q H 1 v A b A i C j D Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Q s s F B y k 0 c u c C C Z p h g 4 F x 0 + O m J U k = " > A A A C H 3 i c b V D L S g M x F M 3 4 r O N r 1 K W b Y B H q w j J T h O q u I K j L i t Y W 2 l I y m d s 2 N J M Z k o x S h 3 6 K G 3 / F j Q s V c d e / M X 0 I 2 n o g c D j n X G 7 u 8 W P O l H b d o b W w u L S 8 s p p Z s 9 c 3 N r e 2 n Z 3 d O x U l k k K F R j y S N Z 8 o 4 E x A R T P N o R Z L I K H P o e r 3 z k d + 9 R 6 k Y p G 4 1 f 0 Y m i H p C N Z m l G g j t Z x i w 4 c O E y k F o U E O 7 B w 9 w h e R f C A y O J b Q S T i R 7 B E C f H l T t h s g g p 9 g y 8 m 6 e X c M P E + 8 K c m i K c o t 5 6 s R R D Q J z T j l R K m 6 5 8 a 6 m R K p G e U w s B u J g p j Q H u l A 3 V B B Q l D N d H z g A B 8 a J c D t S J o n N B 6 r v y d S E i r V D 3 2 T D I n u q l l v J P 7 n 1 R P d P m 2 m T M S J B k E n i 9 o J x z r C o 7 Z w w C R Q z f u G E C q Z + S u m X S I J N R 0 o 2 5 T g z Z 4 8 T y q F / F n e u y 5 k S y f T N j J o H x 2 g H P J Q E Z X Q F S q j C q L o C b 2 g N / R u P V u v 1 o f 1 O Y k u W N O Z P f Q H 1 v A b A i C j D Q = = < / l a t e x i t > (b) Multi-step GSP < l a t e x i t s h a 1 _ b a s e 6 4 = " Y c + m c R o D n l O 5 s X R 6 3 / r 0 r j N G x R g = " > A A A C F n i c b V B N S w M x F M z W r 7 p + V T 1 6 C R a h H i y 7 R V B v B Q 9 6 E S p a W + g u J Z u + t q H Z 7 J J k h b L 0 X 3 j x r 3 j x o O J V v P l v T N s V t H U g M M y 8 4 e V N E H O m t O N 8 W b m F x a X l l f y q v b a + s b l V 2 N 6 5 U 1 E i K d R p x C P Z D I g C z g T U N d M c m r E E E g Y c G s H g f O w 3 7 k E q F o l b P Y z B D 0 l P s C 6 j R B u p X S h 7 A f S Y S C k I D X J k l 4 J D f J V w z Y 6 U h h h f 3 N R s D 0 T n x 2 8 X i k 7 Z m Q D P E z c j R Z S h 1 i 5 8 e p 2 I J q G J U 0 6 U a r l O r P 2 U S M 0 o h 5 H t J Q p i Q g e k B y 1 D B Q l B + e n k r h E + M E o H d y N p n t B 4 o v 5 O p C R U a h g G Z j I k u q 9 m v b H 4 n 9 d K d P f U T 5 m I E w 2 C T h d 1 E 4 5 1 h M c l 4 Q 6 T Q D U f G k K o Z O a v m P a J J N R 0 o G x T g j t 7 8 j y p V 8 p n Z f e 6 U q w e Z 2 3 k 0 R 7 a R y X k o h N U R Z e o h u q I o g f 0 h F 7 Q q / V o P V t v 1 v t 0 N G d l m V 3 0 B 9 b H N 3 W 8 n w Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y c + m c R o D n l O 5 s X R 6 3 / r 0 r j N G x R g = " > A A A C F n i c b V B N S w M x F M z W r 7 p + V T 1 6 C R a h H i y 7 R V B v B Q 9 6 E S p a W + g u J Z u + t q H Z 7 J J k h b L 0 X 3 j x r 3 j x o O J V v P l v T N s V t H U g M M y 8 4 e V N E H O m t O N 8 W b m F x a X l l f y q v b a + s b l V 2 N 6 5 U 1 E i K d R p x C P Z D I g C z g T U N d M c m r E E E g Y c G s H g f O w 3 7 k E q F o l b P Y z B D 0 l P s C 6 j R B u p X S h 7 A f S Y S C k I D X J k l 4 J D f J V w z Y 6 U h h h f 3 N R s D 0 T n x 2 8 X i k 7 Z m Q D P E z c j R Z S h 1 i 5 8 e p 2 I J q G J U 0 6 U a r l O r P 2 U S M 0 o h 5 H t J Q p i Q g e k B y 1 D B Q l B + e n k r h E + M E o H d y N p n t B 4 o v 5 O p C R U a h g G Z j I k u q 9 m v b H 4 n 9 d K d P f U T 5 m I E w 2 C T h d 1 E 4 5 1 h M c l 4 Q 6 T Q D U f G k K o Z O a v m P a J J N R 0 o G x T g j t 7 8 j y p V 8 p n Z f e 6 U q w e Z 2 3 k 0 R 7 a R y X k o h N U R Z e o h u q I o g f 0 h F 7 Q q / V o P V t v 1 v t 0 N G d l m V 3 0 B 9 b H N 3 W 8 n w Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y c + m c R o D n l O 5 s X R 6 3 / r 0 r j N G x R g = " > A A A C F n i c b V B N S w M x F M z W r 7 p + V T 1 6 C R a h H i y 7 R V B v B Q 9 6 E S p a W + g u J Z u + t q H Z 7 J J k h b L 0 X 3 j x r 3 j x o O J V v P l v T N s V t H U g M M y 8 4 e V N E H O m t O N 8 W b m F x a X l l f y q v b a + s b l V 2 N 6 5 U 1 E i K d R p x C P Z D I g C z g T U N d M c m r E E E g Y c G s H g f O w 3 7 k E q F o l b P Y z B D 0 l P s C 6 j R B u p X S h 7 A f S Y S C k I D X J k l 4 J D f J V w z Y 6 U h h h f 3 N R s D 0 T n x 2 8 X i k 7 Z m Q D P E z c j R Z S h 1 i 5 8 e p 2 I J q G J U 0 6 U a r l O r P 2 U S M 0 o h 5 H t J Q p i Q g e k B y 1 D B Q l B + e n k r h E + M E o H d y N p n t B 4 o v 5 O p C R U a h g G Z j I k u q 9 m v b H 4 n 9 d K d P f U T 5 m I E w 2 C T h d 1 E 4 5 1 h M c l 4 Q 6 T Q D U f G k K o Z O a v m P a J J N R 0 o G x T g j t 7 8 j y p V 8 p n Z f e 6 U q w e Z 2 3 k 0 R 7 a R y X k o h N U R Z e o h u q I o g f 0 h F 7 Q q / V o P V t v 1 v t 0 N G d l m V 3 0 B 9 b H N 3 W 8 n w Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y c + m c R o D n l O 5 s X R 6 3 / r 0 r j N G x R g = " > A A A C F n i c b V B N S w M x F M z W r 7 p + V T 1 6 C R a h H i y 7 R V B v B Q 9 6 E S p a W + g u J Z u + t q H Z 7 J J k h b L 0 X 3 j x r 3 j x o O J V v P l v T N s V t H U g M M y 8 4 e V N E H O m t O N 8 W b m F x a X l l f y q v b a + s b l V 2 N 6 5 U 1 E i K d R p x C P Z D I g C z g T U N d M c m r E E E g Y c G s H g f O w 3 7 k E q F o l b P Y z B D 0 l P s C 6 j R B u p X S h 7 A f S Y S C k I D X J k l 4 J D f J V w z Y 6 U h h h f 3 N R s D 0 T n x 2 8 X i k 7 Z m Q D P E z c j R Z S h 1 i 5 8 e p 2 I J q G J U 0 6 U a r l O r P 2 U S M 0 o h 5 H t J Q p i Q g e k B y 1 D B Q l B + e n k r h E + M E o H d y N p n t B 4 o v 5 O p C R U a h g G Z j I k u q 9 m v b H 4 n 9 d K d P f U T 5 m I E w 2 C T h d 1 E 4 5 1 h M c l 4 Q 6 T Q D U f G k K o Z O a v m P a J J N R 0 o G x T g j t 7 8 j y p V 8 p n Z f e 6 U q w e Z 2 3 k 0 R 7 a R y X k o h N U R Z e o h u q I o g f 0 h F 7 Q q / V o P V t v 1 v t 0 N G d l m V 3 0 B 9 b H N 3 W 8 n w Y = < / l a t e x i t > (a) Inverse Model < l a t e x i t s h a 1 _ b a s e 6 4 = " o 2 S z y 8 f N w w R e o G Q S 1 S H G J H n c 3 6 Q = " > A A A C F X i c b V B N S 8 N A F N z 4 W e N X 1 K O X x S L U g y U p g n o r e N G D U M H a Q l P K Z v P a L t 1 s w u 5 G K K G / w o t / x Y s H F a + C N / + N m 7 a C t g 4 s D D N v e P s m S D h T 2 n W / r I X F p e W V 1 c K a v b 6 x u b X t 7 O z e q T i V F O o 0 5 r F s B k Q B Z w L q m m k O z U Q C i Q I O j W B w k f u N e 5 C K x e J W D x N o R 6 Q n W J d R o o 3 U c Y 7 9 A H p M Z B S E B j m y S + Q I X 4 k 8 A f g 6 D o H b P o j w x + 4 4 R b f s j o H n i T c l R T R F r e N 8 + m F M 0 8 j E K S d K t T w 3 0 e 2 M S M 0 o h 5 H t p w o S Q g e k B y 1 D B Y l A t b P x W S N 8 a J Q Q d 2 N p n t B 4 r P 5 O Z C R S a h g F Z j I i u q 9 m v V z 8 z 2 u l u n v W z p h I U g 2 C T h Z 1 U 4 5 1 j P O O c M g k U M 2 H h h A q m f k r p n 0 i C T U d K N u U 4 M 2 e P E / q l f J 5 2 b u p F K s n 0 z Y K a B 8 d o B L y 0 C m q o k t U Q 3 V E 0 Q N 6 Q i / o 1 X q 0 n q 0 3 6 3 0 y u m B N M 3 v o D 6 y P b y i r n u o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o 2 S z y 8 f N w w R e o G Q S 1 S H G J H n c 3 6 Q = " > A A A C F X i c b V B N S 8 N A F N z 4 W e N X 1 K O X x S L U g y U p g n o r e N G D U M H a Q l P K Z v P a L t 1 s w u 5 G K K G / w o t / x Y s H F a + C N / + N m 7 a C t g 4 s D D N v e P s m S D h T 2 n W / r I X F p e W V 1 c K a v b 6 x u b X t 7 O z e q T i V F O o 0 5 r F s B k Q B Z w L q m m k O z U Q C i Q I O j W B w k f u N e 5 C K x e J W D x N o R 6 Q n W J d R o o 3 U c Y 7 9 A H p M Z B S E B j m y S + Q I X 4 k 8 A f g 6 D o H b P o j w x + 4 4 R b f s j o H n i T c l R T R F r e N 8 + m F M 0 8 j E K S d K t T w 3 0 e 2 M S M 0 o h 5 H t p w o S Q g e k B y 1 D B Y l A t b P x W S N 8 a J Q Q d 2 N p n t B 4 r P 5 O Z C R S a h g F Z j I i u q 9 m v V z 8 z 2 u l u n v W z p h I U g 2 C T h Z 1 U 4 5 1 j P O O c M g k U M 2 H h h A q m f k r p n 0 i C T U d K N u U 4 M 2 e P E / q l f J 5 2 b u p F K s n 0 z Y K a B 8 d o B L y 0 C m q o k t U Q 3 V E 0 Q N 6 Q i / o 1 X q 0 n q 0 3 6 3 0 y u m B N M 3 v o D 6 y P b y i r n u o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o 2 S z y 8 f N w w R e o G Q S 1 S H G J H n c 3 6 Q = " > A A A C F X i c b V B N S 8 N A F N z 4 W e N X 1 K O X x S L U g y U p g n o r e N G D U M H a Q l P K Z v P a L t 1 s w u 5 G K K G / w o t / x Y s H F a + C N / + N m 7 a C t g 4 s D D N v e P s m S D h T 2 n W / r I X F p e W V 1 c K a v b 6 x u b X t 7 O z e q T i V F O o 0 5 r F s B k Q B Z w L q m m k O z U Q C i Q I O j W B w k f u N e 5 C K x e J W D x N o R 6 Q n W J d R o o 3 U c Y 7 9 A H p M Z B S E B j m y S + Q I X 4 k 8 A f g 6 D o H b P o j w x + 4 4 R b f s j o H n i T c l R T R F r e N 8 + m F M 0 8 j E K S d K t T w 3 0 e 2 M S M 0 o h 5 H t p w o S Q g e k B y 1 D B Y l A t b P x W S N 8 a J Q Q d 2 N p n t B 4 r P 5 O Z C R S a h g F Z j I i u q 9 m v V z 8 z 2 u l u n v W z p h I U g 2 C T h Z 1 U 4 5 1 j P O O c M g k U M 2 H h h A q m f k r p n 0 i C T U d K N u U 4 M 2 e P E / q l f J 5 2 b u p F K s n 0 z Y K a B 8 d o B L y 0 C m q o k t U Q 3 V E 0 Q N 6 Q i / o 1 X q 0 n q 0 3 6 3 0 y u m B N M 3 v o D 6 y P b y i r n u o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o 2 S z y 8 f N w w R e o G Q S 1 S H G J H n c 3 6 Q = " > A A A C F X i c b V B N S 8 N A F N z 4 W e N X 1 K O X x S L U g y U p g n o r e N G D U M H a Q l P K Z v P a L t 1 s w u 5 G K K G / w o t / x Y s H F a + C N / + N m 7 a C t g 4 s D D N v e P s m S D h T 2 n W / r I X F p e W V 1 c K a v b 6 x u b X t 7 O z e q T i V F O o 0 5 r F s B k Q B Z w L q m m k O z U Q C i Q I O j W B w k f u N e 5 C K x e J W D x N o R 6 Q n W J d R o o 3 U c Y 7 9 A H p M Z B S E B j m y S + Q I X 4 k 8 A f g 6 D o H b P o j w x + 4 4 R b f s j o H n i T c l R T R F r e N 8 + m F M 0 8 j E K S d K t T w 3 0 e 2 M S M 0 o h 5 H t p w o S Q g e k B y 1 D B Y l A t b P x W S N 8 a J Q Q d 2 N p n t B 4 r P 5 O Z C R S a h g F Z j I i u q 9 m v V z 8 z 2 u l u n v W z p h I U g 2 C T h Z 1 U 4 5 1 j P O O c M g k U M 2 H h h A q m f k r p n 0 i C T U d K N u U 4 M 2 e P E / q l f J 5 2 b u p F K s n 0 z Y K a B 8 d o B L y 0 C m q o k t U Q 3 V E 0 Q N 6 Q i / o 1 X q 0 n q 0 3 6 3 0 y u m B N M 3 v o D 6 y P b y i r n u o = < / l a t e x i t > x t+1 < l a t e x i t s h a 1 _ b a s e 6 4 = " I X t y A j 4 N S a M y Z w a x l 3 + f W O M N X Z 8 = " > A A A B 7 X i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S I I Q k m k o N 4 K X j x W M L b Q h r L Z b t q l m 0 3 Y n Y g l 9 E d 4 8 a D i 1 f / j z X / j t s 1 B W x 8 M P N 6 b Y W Z e m E p h 0 H W / n Z X V t f W N z d J W e X t n d 2 + / c n D 4 Y J J M M + 6 z R C a 6 H V L D p V D c R 4 G S t 1 P N a R x K 3 g p H N 1 O / 9 c i 1 E Y m 6 x 3 H K g 5 g O l I g E o 2 i l 1 l M v x 3 N v 0 q t U 3 Z o 7 A 1 k m X k G q U K D Z q 3 x 1 + w n L Y q 6 Q S W p M x 3 N T D H K q U T D J J + V u Z n h K 2 Y g O e M d S R W N u g n x 2 7 o S c W q V P o k T b U k h m 6 u + J n M b G j O P Q d s Y U h 2 b R m 4 r / e Z 0 M o 6 s g F y r N k C s 2 X x R l k m B C p r + T v t C c o R x b Q p k W 9 l b C h l R T h j a h s g 3 B W 3 x 5 m f g X t e u a d 1 e v N u p F G i U 4 h h M 4 A w 8 u o Q G 3 0 A Q f G I z g G V 7 h z U m d F + f d + Z i 3 r j j F z B H 8 g f P 5 A 3 e E j y U = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I X t y A j 4 N S a M y Z w a x l 3 + f W O M N X Z 8 = " > A A A B 7 X i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S I I Q k m k o N 4 K X j x W M L b Q h r L Z b t q l m 0 3 Y n Y g l 9 E d 4 8 a D i 1 f / j z X / j t s 1 B W x 8 M P N 6 b Y W Z e m E p h 0 H W / n Z X V t f W N z d J W e X t n d 2 + / c n D 4 Y J J M M + 6 z R C a 6 H V L D p V D c R 4 G S t 1 P N a R x K 3 g p H N 1 O / 9 c i 1 E Y m 6 x 3 H K g 5 g O l I g E o 2 i l 1 l M v x 3 N v 0 q t U 3 Z o 7 A 1 k m X k G q U K D Z q 3 x 1 + w n L Y q 6 Q S W p M x 3 N T D H K q U T D J J + V u Z n h K 2 Y g O e M d S R W N u g n x 2 7 o S c W q V P o k T b U k h m 6 u + J n M b G j O P Q d s Y U h 2 b R m 4 r / e Z 0 M o 6 s g F y r N k C s 2 X x R l k m B C p r + T v t C c o R x b Q p k W 9 l b C h l R T h j a h s g 3 B W 3 x 5 m f g X t e u a d 1 e v N u p F G i U 4 h h M 4 A w 8 u o Q G 3 0 A Q f G I z g G V 7 h z U m d F + f d + Z i 3 r j j F z B H 8 g f P 5 A 3 e E j y U = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I X t y A j 4 N S a M y Z w a x l 3 + f W O M N X Z 8 = " > A A A B 7 X i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S I I Q k m k o N 4 K X j x W M L b Q h r L Z b t q l m 0 3 Y n Y g l 9 E d 4 8 a D i 1 f / j z X / j t s 1 B W x 8 M P N 6 b Y W Z e m E p h 0 H W / n Z X V t f W N z d J W e X t n d 2 + / c n D 4 Y J J M M + 6 z R C a 6 H V L D p V D c R 4 G S t 1 P N a R x K 3 g p H N 1 O / 9 c i 1 E Y m 6 x 3 H K g 5 g O l I g E o 2 i l 1 l M v x 3 N v 0 q t U 3 Z o 7 A 1 k m X k G q U K D Z q 3 x 1 + w n L Y q 6 Q S W p M x 3 N T D H K q U T D J J + V u Z n h K 2 Y g O e M d S R W N u g n x 2 7 o S c W q V P o k T b U k h m 6 u + J n M b G j O P Q d s Y U h 2 b R m 4 r / e Z 0 M o 6 s g F y r N k C s 2 X x R l k m B C p r + T v t C c o R x b Q p k W 9 l b C h l R T h j a h s g 3 B W 3 x 5 m f g X t e u a d 1 e v N u p F G i U 4 h h M 4 A w 8 u o Q G 3 0 A Q f G I z g G V 7 h z U m d F + f d + Z i 3 r j j F z B H 8 g f P 5

A 3 e E j y U = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I X t y A j 4 N S a M y Z w a x l 3

+ f W O M N X Z 8 = " > A A A B 7 X i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S I I Q k m k o N 4 K X j x W M L b Q h r L Z b t q l m 0 3 Y n Y g l 9 E d 4 8 a D i 1 f / j z X / j t s 1 B W x 8 M P N 6 b Y W Z e m E p h 0 H W / n Z X V t f W N z d J W e X t n d 2 + / c n D 4 Y J J M M + 6 z R C a 6 H V L D p V D c R 4 G S t 1 P N a R x K 3 g p H N 1 O / 9 c i 1 E Y m 6 x 3 H K g 5 g O l I g E o 2 i l 1 l M v x 3 N v 0 q t U 3 Z o 7 A 1 k m X k G q U K D Z q 3 x 1 + w n L Y q 6 Q S W p M x 3 N T D H K q U T D J J + V u Z n h K 2 Y g O e M d S R W N u g n x 2 7 o S c W q V P o k T b U k h m 6 u + J n M b G j O P Q d s Y U h 2 b R m 4 r / e Z 0 M o 6 s g F y r N k C s 2 X x R l k m B C p r + T v t C c o R x b Q p k W 9 l b C h l R T h j a h s g 3 B W 3 x 5 m f g X t e u a d 1 e v N u p F G i U 4 h h M 4 A w 8 u o Q G 3 0 A Q f G I z g G V 7 h z U m d F + f d + Z i 3 r j j F z B H 8 g f P 5

A 3 e E j y U = < / l a t e x i t > inferred from the observations. The agent then infers how to perform the task (i.e., plan for imitation) using this state. Unfortunately, computer vision systems are often unable to estimate the state variables accurately and it has proven non-trivial for downstream planning systems to be robust to such errors.

In this paper, we follow (Agrawal et al., 2016; Levine et al., 2016; Pinto & Gupta, 2016) in pursuing an alternative paradigm, where an agent explores the environment without any expert supervision and distills this exploration data into goal-directed skills. These skills can then be used to imitate the visual demonstration provided by the expert (Nair et al., 2017) . Here, by skill we mean a function that predicts the sequence of actions to take the agent from the current observation to the goal. We call this function a goal-conditioned skill policy (GSP). The GSP is learned in a self-supervised way by re-labeling the states visited during the agent's exploration of the environment as goals and the actions executed by the agent as the prediction targets, similar to (Agrawal et al., 2016; Andrychowicz et al., 2017) . During inference, given goal observations from a demonstration, the GSP can infer how to reach these goals in turn from the current observation, and thereby imitate the task step-by-step.

One critical challenge in learning the GSP is that, in general, there are multiple possible ways of going from one state to another: that is, the distribution of trajectories between states is multimodal. We address this issue with our novel forward consistency loss based on the intuition that, for most tasks, reaching the goal is more important than how it is reached. To operationalize this, we first learn a forward model that predicts the next observation given an action and a current observation. We use the difference in the output of the forward model for the GSP-selected action and the ground truth next state to train the GSP. This loss has the effect of making the GSP-predicted action consistent with the ground-truth action instead of exactly matching the actions themselves, thus ensuring that actions that are different from the ground-truth-but lead to the same next stateare not inadvertently penalized. To account for varying number of steps required to reach different goals, we propose to jointly optimize the GSP with a goal recognizer that determines if the current goal has been satisfied. See Figure 1 for a schematic illustration of the GSP architecture.

We call our method zero-shot because the agent never has access to expert actions, neither during training of the GSP nor for task demonstration at inference. In contrast, most recent work on oneshot imitation learning requires full knowledge of actions and a wealth of expert demonstrations during training (Duan et al., 2017; Finn et al., 2017) . In summary, we propose a method that (1) does not require any extrinsic reward or expert supervision during learning, (2) only needs demonstrations during inference, and (3) restricts demonstrations to visual observations alone rather than full stateactions. Instead of learning by imitation, our agent learns to imitate.

We evaluate our zero-shot imitator on real-world robots for rope manipulation tasks using a Baxter and office navigation using a TurtleBot. We show that the proposed forward consistency loss improves the performance on the complex task of knot tying from 36% to 60% accuracy. In navigation experiments, we steer a simple wheeled robot around partially-observable office environments and show that the learned GSP generalizes to unseen environments. Furthermore, using navigation experiments in VizDoom environment, we show that (GSP) learned using curiosity-driven exploration (Oudeyer et al., 2007; Pathak et al., 2017; Schmidhuber, 1991) can more accurately follow demonstrations as compared to using random exploration data for learning the GSP. Overall our experiments show that the forward-consistent GSP can be used to imitate a variety of tasks without making environment or task-specific assumptions.

2. LEARNING TO IMITATE WITHOUT EXPERT SUPERVISION:

Let S : {x 1 , a 1 , x 2 , a 2 , ..., x T } be the sequence of observations and actions generated by the agent as it explores its environment using the policy a = π E (s). This exploration data is used to learn the goal-conditioned skill policy (GSP) π takes as input a pair of observations (x i , x g ) and outputs sequence of actions ( a τ : a 1 , a 2 ...a K ) required to reach the goal observation (x g ) from the current observation (x i ).

EQUATION

where states x i , x g are sampled from the S. The number of actions, K, is also inferred by the model. We represent π by a deep network with parameters θ π in order to capture complex mappings from visual observations (x) to actions. π can be thought of as a variable-step generalization of the inverse dynamics model (Jordan & Rumelhart, 1992), or as the policy corresponding to a universal value function (Foster & Dayan, 2002; Schaul et al., 2015) , with the difference that x g need not be the end goal of a task but can also be an intermediate sub-goal.

Let the task to be imitated be provided as a sequence of images D : {x d 1 , x d 2 , ..., x d N } captured when the expert demonstrates the task. This sequence of images D could either be temporally dense or sparse. Our agent uses the learned GSP π to imitate the sequence of visual observations D starting from its initial state x 0 by following actions predicted by π(x 0 , x d 1 ; θ π ). Let the observation after executing the predicted action be x 0 . Since multiple actions might be required to reach close to x d 1 , the agent queries a separate goal recognizer network to ascertain if the current observation is close to the goal or not. If the answer is negative, the agent executes the action a = π(x 0 , x d 1 ; θ π ). This process is repeated iteratively until the goal recognizer outputs that agent is near the goal, or a maximum number of steps are reached. Let the observation of the agent at this point be x1 . After reaching close to the first observation (x d 1 ) in the demonstration, the agent sets its goal as (x d 2 ) and repeats the process. The agent stops when all observations in the demonstrations are processed.

Note that in the method of imitation described above, the expert is never required to convey to the agent what actions it performed. In the following subsections we describe how we learn the GSP, forward consistency loss, goal recognizer network and various baseline methods.

2.1. LEARNING THE GOAL-CONDITIONED SKILL POLICY (GSP):

We first describe the one-step version of GSP and then extend it to variable length multi-step skills. One-step trajectories take the form of (x t , a t , x t+1 ) and GSP, ât = π(x t , x t+1 ; θ π ), is trained by minimizing the standard cross-entropy loss L(a t , ât ),

EQUATION

with respect to parameters θ π , where p and ât are the ground-truth and predicted action distributions. While we do not have access to true p, we empirically approximate it using samples from the distribution, a t , that are executed by the agent during exploration. For minimizing the crossentropy loss, it is common to assume p as a delta function at a t . However, this assumption is notably violated if p is inherently multi-modal and high-dimensional. If we optimize say a deep neural network assuming p to be a delta function, the same inputs will be presented with different targets (due to multi-modality) leading to high-variance in gradients which in turn would make learning challenging.

In our setup, such multi-modality can occur because multiple actions can lead the agent to the same future observation from the initial observation. For instance, in navigation, if the agent is stuck against a corner, turning or moving forward all collapse to the same effect. The issue of multimodality becomes more critical as the length of trajectories grow, because more and more paths may take the agent from the initial observation to the goal observation given more time. Furthermore, it would require many samples to even obtain a good empirical estimate of a high-dimensional multimodal action distribution p.

2.2. FORWARD CONSISTENCY LOSS:

One way to account for multi-modality is by employing the likes of variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014). However, in many practical situations it is not feasible to obtain ample data for each mode. In this work, we propose an alternative based on the insight that in many scenarios, we only care about whether the agent reached the final state or not and the exact trajectory is of lesser interest. Instead of penalizing the actions predicted by the GSP to match the ground truth, we propose to learn the parameters of GSP by minimizing the distance between observation xt+1 resulting by executing the predicted action ât = π(x t , x t+1 ; θ π ) and the observation x t+1 , which is the result of executing the ground truth action a t being used to train the GSP. In this formulation, even if the predicted and ground-truth action are different, the predicted action will not be penalized if it leads to the same next state as the ground-truth action. While this formulation will not explicitly maintain all modes of the action distribution, it will reduce the variance in gradients and thus help learning. We call this penalty the forward consistency loss.

Note that it is not immediately obvious as to how to operationalize forward consistency loss for two reasons: (a) we need the access to a good forward dynamics model that can reliably predict the effect of an action (i.e., the next observation state) given the current observation state, and (b) such a dynamics model should be differentiable in order to train the GSP using the state prediction error. Both of these issues could be resolved if an analytic formulation of forward dynamics is known.

In many scenarios of interest, especially if states are represented as images, an analytic forward model is not available. In this work, we learn the forward dynamics f model from the data, and is defined as xt+1 = f (x t , a t ; θ f ). Let xt+1 = f (x t , ât ; θ f ) be the state prediction for the action predicted by π. Because the forward model is not analytic and learned from data, in general, there is no guarantee that xt+1 = xt+1 , even though executing these two actions, a t , ât , in the real-world will have the same effect. In order to make the outcome of action predicted by the GSP and the ground-truth action to be consistent with each other, we include an additional term,

x t+1 -xt+1 2 2

in our loss function and infer the parameters θ f by minimizing x t+1 -xt+1

2 2 + λ x t+1 - xt+1 2 2

, where λ is a scalar hyper-parameter. The first term ensures that the learned forward model explains ground truth transitions (x t , a t , x t+1 ) collected by the agent and the second term ensures consistency. The joint objective for training GSP with forward model consistency is:

EQUATION

s.t. xt+1 = f (x t , a t ; θ f ) xt+1 = f (x t , ât ; θ f ) ât = π(x t ,

x t+1 ; θ π ) Note that learning θ π , θ f jointly from scratch is precarious, because the forward model f might not be good in the beginning, and hence could make the gradient updates noisier for π. To address this issue, we first pre-train the forward model with only the first term and GSP separately by blocking the gradient flow and then fine-tune jointly.

Generalization to feature space dynamics Past work has shown that learning forward dynamics in the feature space as opposed to raw observation space is more robust and leads to better generalization (Agrawal et al., 2016; Pathak et al., 2017) . Following these works, we extend the GSP to make predictions in feature representation φ(x t ), φ(x t+1 ) of the observations x t , x t+1 respectively learned through the self-supervised task of action prediction. The forward consistency loss is then computed by making predictions in this feature space φ instead of raw observations. The optimization objective for feature space generalization with mutli-step objective is shown in Equation ( 4).

Generalization to multi-step GSP We extend our one-step optimization to variable length sequence of actions in a straightforward manner by having a multi-step GSP π m model with a step-wise forward consistency loss. The GSP π m maintains an internal recurrent memory of the system and outputs actions conditioned on current observation x t , starting from x i to reach goal observation x T . The forward consistency loss is computed at each time step, and jointly optimized with the action prediction loss over the whole trajectory. The final multi-step objective with feature space dynamics is as follows:

EQUATION

s.t. φ(x t+1 ) = f (φ(x t ), a t ; θ f ) φ(x t+1 ) = f (φ(x t ), ât ; θ f ) ât = π(φ(x t ), φ(x T ); θ π ) where φ(.

) is represented by a CNN with parameters θ φ . The number of steps taken by the multistep GSP π m to reach the goal at inference is variable depending on the decision of goal recognizer; described in next subsection. Note that, in this objective, if φ is identity then the dynamics simply reduces to modeling in raw observation space. We analyze feature space prediction in VizDoom 3D navigation and stick to observation space in the rope manipulation and the office navigation tasks.

The multi-step forward-consistent GSP π m is implemented using a recurrent network which at every step takes as input the feature representation of the current (φ(x t )) state, goal (φ(x T )) states, action at the previous time step (a t-1 ) and the internal hidden representation h t-1 of the recurrent units and predicts ât . Note that inputting the previous action to GSP π m at each time step could be redundant given that hidden representation is already maintaining a history of the trajectory. Nonetheless, it is helpful to explicitly model this history. This formulation amounts to building an auto-regressive model of the joint action that estimates probability P (a t |x 1 , a 1 , ...a t-1 , x t , x g ) at every time step. It is possible to further extend our forward-consistent GSP π m to build multi-step forward model, but we leave that direction of future work.

2.3. GOAL RECOGNIZER:

We train a goal recognizer network to figure out if the current goal is reached and therefore allow the agent to take variable numbers of steps between goals. Goal recognition is especially critical when the agent has to transit through a sequence of intermediate goals, as is the case for visual imitation, as otherwise compounding error could quickly lead to divergence from the demonstration. This recognition is simple given knowledge of the true physical state, but difficult when working with visual observations. Aside from the usual challenges of visual recognition, the dependence of observations on the agent's own dynamics further complicates goal recognition, as the same goal can appear different while moving forward or turning during navigation.

We pose goal recognition as a binary classification problem that given an observation x i and the goal x g infers if x i is close to x g or not. Lacking expert supervision of goals, we draw goal observations at random from the agent's experience during exploration, since they are known to be feasible. For each such pseudo-goal, we consider observations that were only a few actions away to be positives (i.e., close to the goal) and the remaining observations that were more than a fixed number of actions (i.e., a margin) away as negatives. We trained the goal classifier using the standard cross-entropy loss. Like the skill policy, our goal recognizer is conditioned on the goal for generalization across goals. We found that training an independent goal recognition network consistently outperformed the alternative approach that augments the action space with a "stop" action. Making use of temporal proximity as supervision has also been explored for feature learning in the concurrent work of Sermanet et al. (2018).

2.4. ABLATIONS AND BASELINES:

Our proposed formulation of GSP composed of following components: (a) recurrent variable-length skill policy network, (b) explicitly encoding previous action in the recurrence, (c) goal recognizer, (d) forward consistency loss function, and (w) learning forward dynamics in the feature space instead of raw observation space. We systematically ablate these components of forward-consistent GSP, to quantitatively review the importance of each component and then perform comparisons to the prior approaches that could be deployed for the task of visual imitation. 2017) leverage vanilla inverse dynamics to follow demonstration in rope manipulation setup. We compare to their method in both visual navigation and manipulation. (3) GSP-NoPrevAction-NoFwdConst is the ablation of our recurrent GSP without previous action history and without forward consistency loss. ( 4) GSP-NoFwdConst refers to our recurrent GSP with previous action history, but without forward consistency objective. ( 5) GSP-FwdRegularizer refers to the model where forward prediction is only used to regularize the features of GSP but has no role to play in the loss function of predicted actions. The purpose of this variant is to particularly ablate the benefit of consistency loss function with respect to just having forward model as feature regularizer. ( 6) GSP refers to our complete method with all the components. We now discuss the experiments and evaluate these baselines.

3. EXPERIMENTS:

We evaluate our model by testing its performance on: rope manipulation using Baxter robot, navigation of a wheeled robot in cluttered office environments, and simulated 3D navigation. The key requirements of a good skill policy are that it should generalize to unseen environments and new goals while staying robust to irrelevant distractors in the observations. For rope manipulation, we evaluate generalization by testing the ability of the robot to manipulate the rope into configurations such as knots that were not seen during random exploration. For navigation, both real-world and simulation, we check generalization by testing on a novel building/floor.

3.1. ROPE MANIPULATION:

Manipulation of non-rigid and deformable objects, e.g., rope, is a challenging problem in robotics. Even humans learn complex rope manipulation such as tying knots, either by observing an expert Published as a conference paper at ICLR 2018

Method:

Success %

Inverse Model [Nair et.al. 2017 ] 36% ± 9.6% Forward-regularized GSP 44% ± 9.9% Forward-consistent GSP [Ours] 60% ± 9.8%

Table 1 : GSP trained using forward consistency loss, significantly outperforms the baseline GSP model (i.e. inverse model only) at knot tying.

Manipulation of non-rigid and deformable objects, e.g., rope, is a challenging problem in robotics.

Even humans learn complex rope manipulation such as tying knots, either by observing an expert perform it or by receiving explicit instructions. To test whether our agent could manipulate ropes by simply observing a human, we use the data collected by Nair et al. (2017) , where a Baxter robot manipulated a rope kept on the table in front of it. During exploration, the robot interacts with the rope by using a pick and place primitive that chooses a random point on the rope, and displaces it by a randomly chosen length and direction. This process is repeated number of times to collect about 60K interaction pairs of the form (x t , a t , x t+1 ) that are used to train the GSP.

During inference, our proposed approach is tasked to follow a visual demonstration provided by a human expert for manipulating the rope into a complex 'S' shape and tying a knot. Our agent, Baxter robot, only gets to observe the image sequence of intermediate states, as human manipulates the rope, without any access to the corresponding actions. Note that the knot shape is never encountered during the self-supervised data collection phase and therefore the learned GSP model would have to generalize to be able to follow the human demonstration. More details follow in the supplementary material, Section A.

Metric:

The performance of the model is evaluated by measuring the non-rigid registration cost between the rope state achieved by the robot and the state demonstrated by the human at every step in the demonstration. The matching cost is measured using the thin plate spline robust point matching technique (TPS-RPM) described in (Chui & Rangarajan, 2003) . While TPS-RPM provides a good metric for measuring performance for constructing the 'S' shape, it is not an appropriate metric for knots because the configuration of the rope in a knot is 3D due to intertwining of the rope, and fails to find the correct point correspondences. We, therefore, use accuracy as the metric in knot tying where the completion of successful knot is judged by human verification.

Visual Imitation We compare our approach to the previous best method of visual imitation that deploys an inverse model which takes as input a pair of current and goal images to output the desired action to reach goal (Nair et al., 2017) . We re-implement the baseline and train in our setup for a fair comparison. The comparison is performed apples-to-apples and the only difference is that we deploy forward-consistency loss for our approach. The results in Figure 2 show that our method significantly outperforms the baseline at task of manipulating the rope in the 'S' shape and achieves an accuracy of 60% in comparison to 36% achieved by the baseline.

3.2. NAVIGATION IN INDOOR OFFICE ENVIRONMENTS:

A natural way to instruct a robot to move in an indoor office environment is to ask it to go near a certain location, such as, a refrigerator or a certain person's office. Instead of using language to command the robot, in this work, we communicate with the robot by either showing it a single image of the goal, or a sequence of images leading to faraway goals. In both scenarios, the robot is required to autonomously determine the motor commands for moving to the goal. We used TurtleBot2 for navigation using an onboard camera for sensing RGB images. For learning the GSP, an automated self-supervised scheme for data collection was devised that required no human supervision. The robot collected number of trajectories that contain 230K interactions data, i.e. (x t , a t , x t+1 ), from two floors of a academic building in total. We then deployed the learned model on a separate floor of a building with substantially different textures and furniture layout for performing visual imitation at test time. The details of robotic setup, data collection and network architecture of GSP are described in supplementary material, Section A.

7:

(a) TPS-RPM error for 'S' shape manipulation perform it or by receiving explicit instructions. We test whether our agent could manipulate ropes by simply observing a human perform it. We use the data collected by Nair et al. (2017) , where a Baxter robot manipulated a rope kept on the table in front of it. During exploration, the robot interacts with the rope by using a pick and place primitive that chooses a random point on the rope and displaces it by a randomly chosen length and direction. This process is repeated a number of times to collect about 60K interaction pairs of the form (x t , a t , x t+1 ) that are used to train the GSP.

< l a t e x i t s h a _ b a s e = " + M w V k N O A H H m y g Y Y u F U p I d Q = " > A A A C F H i c b V C S g N B F J G e M r a m k z G E Q F D b s a h e w s R F W k z V C E v T u K Z n J d Z m a F E P I T N v K j Y W K r Y W d f + M k c L X g Y H D O e c y w F V w b / J i a n p m d m C P F + Y X F p e X S y u q F T j L F M G C J S N R l C B o F l x g Y b g R e p g o h D g U w t v j o d + Q V I u u m l I h h v J I A W O m q t L s N O T u / b O / V O K S i W K R v Z d a o k K K N A b J z k b J b c U e g f m X k z L J V + V P l q d h G U x S s M E a N N S + A M Z w I H x V a m M Q V C z f Y t F R C j L r d H o J t W Y y i R J p E j P t G H W O t e H N p k D K a r f t D T + v m Z n o s N n M s M S j b + K M o E N Q k d V k Q X C E z o m c J M M X t r p R Q Q E z t s i i L c H f f J f E u x X j i r e X u Z t F M g S D b x C M H p E p O i E C w s g e S T P M V c J c V + d t H J w p k g P O + x c L B V < / l a t e x i t > < l a t e x i t s h a _ b a s e = " + M w V k N O A H H m y g Y Y u F U p I d Q = " > A A A C F H i c b V C S g N B F J G e M r a m k z G E Q F D b s a h e w s R F W k z V C E v T u K Z n J d Z m a F E P I T N v K j Y W K r Y W d f + M k c L X g Y H D O e c y w F V w b / J i a n p m d m C P F + Y X F p e X S y u q F T j L F M G C J S N R l C B o F l x g Y b g R e p g o h D g U w t v j o d + Q V I u u m l I h h v J I A W O m q t L s N O T u / b O / V O K S i W K R v Z d a o k K K N A b J z k b J b c U e g f m X k z L J V + V P l q d h G U x S s M E a N N S + A M Z w I H x V a m M Q V C z f Y t F R C j L r d H o J t W Y y i R J p E j P t G H W O t e H N p k D K a r f t D T + v m Z n o s N n M s M S j b + K M o E N Q k d V k Q X C E z o m c J M M X t r p R Q Q E z t s i i L c H f f J f E u x X j i r e X u Z t F M g S D b x C M H p E p O i E C w s g e S T P M V c J c V + d t H J w p k g P O + x c L B V < / l a t e x i t > < l a t e x i t s h a _ b a s e = " + M w V k N O A H H m y g Y Y u F U p I d Q = " > A A A C F H i c b V C S g N B F J G e M r a m k z G E Q F D b s a h e w s R F W k z V C E v T u K Z n J d Z m a F E P I T N v K j Y W K r Y W d f + M k c L X g Y H D O e c y w F V w b / J i a n p m d m C P F + Y X F p e X S y u q F T j L F M G C J S N R l C B o F l x g Y b g R e p g o h D g U w t v j o d + Q V I u u m l I h h v J I A W O m q t L s N O T u / b O / V O K S i W K R v Z d a o k K K N A b J z k b J b c U e g f m X k z L J V + V P l q d h G U x S s M E a N N S + A M Z w I H x V a m M Q V C z f Y t F R C j L r d H o J t W Y y i R J p E j P t G H W O t e H N p k D K a r f t D T + v m Z n o s N n M s M S j b + K M o E N Q k d V k Q X C E z o m c J M M X t r p R Q Q E z t s i i L c H f f J f E u x X j i r e X u Z t F M g S D b x C M H p E p O i E C w s g e S T P M V c J c V + d t H J w p k g P O + x c L B V < / l a t e x i t > < l a t e x i t s h a _ b a s e = " + M w V k N O A H H m y g Y Y u F U p I d Q = " > A A A C F H i c b V C S g N B F J G e M r a m k z G E Q F D b s a h e w s R F W k z V C E v T u K Z n J d Z m a F E P I T N v K j Y W K r Y W d f + M k c L X g Y H D O e c y w F V w b / J i a n p m d m C P F + Y X F p e X S y u q F T j L F M G C J S N R l C B o F l x g Y b g R e p g o h D g U w t v j o d + Q V I u u m l I h h v J I A W O m q t L s N O T u / b O / V O K S i W K R v Z d a o k K K N A b J z k b J b c U e g f m X k z L J V + V P l q d h G U x S s M E a N N S + A M Z w I H x V a m M Q V C z f Y t F R C j L r d H o J t W Y y i R J p E j P t G H W O t e H N p k D K a r f t D T + v m Z n o s N n M s M S j b + K M o E N Q k d V k Q X C E z o m c J M M X t r p R Q Q E z t s i i L c H f f J f E u x X j i r e X u Z t F M g S D b x C M H p E p O i E C w s g e S T P M V c J c V + d t H J w p k g P O + x c L B V < / l a t e x i t > (b) Success rate for Knot-tying < l a t e x i t s h a 1 _ b a s e 6 4 = " c B f 7 Q n 6 Y b k K I P 8 M i v M m r J 7 4 h T 5 M = " > A A A C B 3 i c b V C 7 T g J B F J 3 F F + J r 1 d L C i c Q E C 8 k u j d q R 2 J j Y Y B Q h A U J m h 7 s w Y X Z m M 3 P X h B B K G 3 / F x k K N r b 9 g 5 9 + 4 P A o F T 3 V y z r 2 5 5 5 4 g l s K i 5 3 0 7 m a X l l d W 1 7 H p u Y 3 N r e 8 f d 3 b u 3 O j E c q l x L b e o B s y C F g i o K l F C P D b A o k F A L + p d j v / Y A x g q t 7 n A Q Q y t i X S V C w R m m U t s 9 L A Q n 9 D b h H K y l h i H Q U B t 6 r T S e 4 k C o b t v N e 0 V v A r p I / B n J k x k q b f e r 2 d E 8 i U A h l 8 z a h u / F 2 B o y g 4 J L G O W a i Y W Y 8 T 7 r Q i O l i k V g W 8 P J I y N 6 n C q d S Y J Q K 6 Q T 9 f f G k E X W D q I g n Y w Y 9 u y 8 N x b / 8 x o J h u e t o V B x g q D 4 9 F C Y S I q a j l u h H W G A o x y k h H E j 0 q y U 9 5 h h H N P u c m k J / v z L i 6 R a K l 4 U / Z t S v u z N 2 s i S A 3 J E C s Q n Z 6 R M r k i F V A k n j + S Z v J I 3 5 8 l 5 c d 6 d j + l o x p n t 7 J M / c D 5 / A J G G m J Q = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c B f 7 Q n 6 Y b k K I P 8 M i v M m r J 7 4 h T 5 M = " > A A A C B 3 i c b V C 7 T g J B F J 3 F F + J r 1 d L C i c Q E C 8 k u j d q R 2 J j Y Y B Q h A U J m h 7 s w Y X Z m M 3 P X h B B K G 3 / F x k K N r b 9 g 5 9 + 4 P A o F T 3 V y z r 2 5 5 5 4 g l s K i 5 3 0 7 m a X l l d W 1 7 H p u Y 3 N r e 8 f d 3 b u 3 O j E c q l x L b e o B s y C F g i o K l F C P D b A o k F A L + p d j v / Y A x g q t 7 n A Q Q y t i X S V C w R m m U t s 9 L A Q n 9 D b h H K y l h i H Q U B t 6 r T S e 4 k C o b t v N e 0 V v A r p I / B n J k x k q b f e r 2 d E 8 i U A h l 8 z a h u / F 2 B o y g 4 J L G O W a i Y W Y 8 T 7 r Q i O l i k V g W 8 P J I y N 6 n C q d S Y J Q K 6 Q T 9 f f G k E X W D q I g n Y w Y 9 u y 8 N x b / 8 x o J h u e t o V B x g q D 4 9 F C Y S I q a j l u h H W G A o x y k h H E j 0 q y U 9 5 h h H N P u c m k J / v z L i 6 R a K l 4 U / Z t S v u z N 2 s i S A 3 J E C s Q n Z 6 R M r k i F V A k n j + S Z v J I 3 5 8 l 5 c d 6 d j + l o x p n t 7 J M / c D 5 / A J G G m J Q = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c B f 7 Q n 6 Y b k K I P 8 M i v M m r J 7 4 h T 5 M = " > A A A C B 3 i c b V C 7 T g J B F J 3 F F + J r 1 d L C i c Q E C 8 k u j d q R 2 J j Y Y B Q h A U J m h 7 s w Y X Z m M 3 P X h B B K G 3 / F x k K N

F A L + p d j v / Y A x g q t 7 n A Q Q y t i X S V C w R m m U t s 9 L A Q n 9 D b h H K y l h i H Q U B t 6 r T S e 4 k C o b t v N e 0 V v A r p I / B n J k x k q b f e r 2 d E 8 i U A h l 8 z a h u / F 2 B o y g 4 J L G O W a i Y W Y 8 T 7 r Q i O l i k V g W 8 P J I y N 6 n C q d S Y J Q K 6 Q T 9 f f G k E X W D q I g n Y w Y 9 u y 8 N x b / 8 x o J h u e t o V B x g q D 4 9 F C Y S I q a j l u h H W G A o x y k h H E j 0 q y U 9 5 h h H N P u c m k J / v z L i 6 R a K l 4 U / Z t S v u z N 2 s i S A 3 J E C s Q n Z 6 R M r k i F V A k n j + S Z

I P 8 M i v M m r J 7 4 h T 5 M = " > A A A C B 3 i c b V C 7 T g J B F J 3 F F + J r 1 d L C i c Q E C 8 k u j d q R 2 J j Y Y B Q h A U J m h 7 s w Y X Z m M 3 P X h B B K G 3 / F x k K N

F A L + p d j v / Y A x g q t 7 n A Q Q y t i X S V C w R m m U t s 9 L A Q n 9 D b h H K y l h i H Q U B t 6 r T S e 4 k C o b t v N e 0 V v A r p I / B n J k x k q b f e r 2 d E 8 i U A h l 8 z a h u / F 2 B o y g 4 J L G O W a i Y W Y 8 T 7 r Q i O l i k V g W 8 P J I y N 6 n C q d S Y J Q K 6 Q T 9 f f G k E X W D q I g n Y w Y 9 u y 8 N x b / 8 x o J h u e t o V B x g q D 4 9 F C Y S I q a j l u h H W G A o x y k h H E j 0 q y U 9 5 h h H N P u c m k J / v z L i 6 R a K l 4 U / Z t S v u z N 2 s i S A 3 J E C s Q n Z 6 R M r k i F V A k n j + S Z

During inference, our proposed approach is tasked to follow a visual demonstration provided by a human expert for manipulating the rope into a complex 'S' shape and tying a knot. Our agent, Baxter robot, only gets to observe the image sequence of intermediate states, as human manipulates the rope, without any access to the corresponding actions. Note that the knot shape is never encountered during the self-supervised data collection phase and therefore the learned GSP model would have to generalize to be able to follow the human demonstration. More details follow in the supplementary material, Section A.1.

The performance of the model is evaluated by measuring the non-rigid registration cost between the rope state achieved by the robot and the state demonstrated by the human at every step in the demonstration. The matching cost is measured using the thin plate spline robust point matching technique (TPS-RPM) described in (Chui & Rangarajan, 2003) . While TPS-RPM provides a good metric for measuring performance for constructing the 'S' shape, it is not an appropriate metric for knots because the configuration of the rope in a knot is 3D due to intertwining of the rope, and it fails to find the correct point correspondences. We, therefore, use success rate as the metric in knot tying where the completion of a successful knot is judged by human verification.

Visual Imitation:

Qualitative examples of our agent trying to manipulate rope are shown in Figure 2 . We compare our approach to the baseline that deploys an inverse model which takes as input a pair of current and goal images to output the desired action to reach the goal (Nair et al., 2017) . We reimplement the baseline and train in our setup for a fair comparison. To further ablate the importance of consistency loss, we compare to a baseline that just uses a forward model as a regularizer of features. The results in Figure 3 show that our method significantly outperforms the baseline at task of manipulating the rope in the 'S' shape and achieves a success rate of 60% in comparison to 36% achieved by the baseline.

A natural way to instruct a robot to move in an indoor office environment is to ask it to go near a certain location, such as a refrigerator or a someone's office. Instead of using language to command the robot, in this work, we communicate with the robot by either showing it a single image of the goal, or a sequence of images leading to faraway goals. In both scenarios, the robot is required to autonomously determine the motor commands for moving to the goal. We used TurtleBot2 for navigation using an onboard camera for sensing RGB images. For learning the GSP, an automated Published as a conference paper at ICLR 2018

Step-7

Step-14

Step-21

Step-28

Step-35

Step-42

Step-44

Step-51

Initial Image:

Final Image (Step-59) Fixed Goal Image Table 1 : Quantitative evaluation of various methods on the task of navigating using a single image of goal in an unseen environment. Each column represents a different run of our system for a different initial/goal image pair. Our full GSP model takes longer to reach the goal on average given a successful run but reaches the goal successfully at a much higher rate.

self-supervised scheme for data collection was devised that doesn't require human supervision. The robot collected a number of navigation trajectories from two floors of a academic building which in total contain 230K interactions data, i.e. (x t , a t , x t+1 ). We then deployed the learned model on a separate floor of a building with substantially different textures and furniture layout for performing visual imitation at test time. The details of the robotic setup, data collection, and network architecture of GSP are described in supplementary material, Section A.2.

1) Goal Finding:

We first tested if the GSP learned by the TurtleBot can enable it to find its way to a goal that is within the same room from just a single image of the goal. To test the extrapolative generalization, we keep the Turtlebot approximately 20-30 steps away from the target location in a way that current and goal observations have no overlap as shown in Figure 4 . We test the robot in an indoor office environment on a different floor that it has never encountered before. We judge the robot to be successful if it stops close to the goal and failure if it crashed into furniture or does not reach the goal within 200 steps. Since the initial and goal images have no overlap, classical techniques such as structure from motion that rely on feature matching cannot be used to infer the executed action. Therefore, in order to reach the goal, the robot must explore its surroundings. We find that our GSP model outperforms the baseline models in reaching the target location. Our model learns the exploratory behavior of rotating in place until it encounters an overlap between its current and goal image. Results are shown in Table 1 and videos are available at the website1 .

2) Visual Imitation In the previous paragraph, we saw that the robot can reach a goal that's within the same room. However, our agent is unable to reach far away goals such as in other rooms using just a single image. In such scenarios, an expert might communicate instructions like go to the door, turn right, go to the closest chair etc. Instead of language instruction, in our setup we provide a sequence of landmark images to convey the same high-level idea. These landmark images were captured from the robot's camera as the expert moved the robot from the start to a goal location. However, note that it is not necessary for the expert to control the robot to capture the images because we don't make use of the expert's actions, but only the images. Instead of providing the image after every action in the demonstration, we only provided every fifth image. The rationale behind this choice is that we want to sample the demonstration sparsely to minimize the agent's Table 3 : Quantitative evaluation of our proposed GSP and the baseline models at following visual demonstrations in VizDoom 3D Navigation. Medians and 95% confidence intervals are reported for demonstration completion and efficiency over 50 seeds and 5 human paths per environment type.

In VizDoom, we collect data by deploying two types of exploration methods: random exploration and curiosity-driven exploration (Pathak et al., 2017) . The hypothesis is that if the initial data collected by the robot is driven by a better strategy than just random, this should eventually help the agent follow long demonstrations better. Our environment consists of 2 maps in total. We train on one map with 5 different starting positions for collecting exploration data. For validation, we collect 5 human demonstrations in a map with the same layout as in training but with different textures.

For zero-shot generalization, we collect 5 human demonstrations in a novel map layout with novel textures. Exact details for data collection and training setup are in the supplementary, Section A.3.

We report the median of maximum distance reached by the robot in following the given sequence of demonstration images. The maximum distance reached is the distance of farthest landmark point that the agent reaches contiguously, i.e., without missing any intermediate landmarks.

Measuring the farthest landmark reached does not capture how efficiently it is reached. Hence, we further measure efficiency of the agent as the ratio of number of steps taken by the agent to reach farthest contiguous landmark with respect to the number of steps shown in human demonstrations.

The task here is same as the one in real robot navigation where the agent is shown a sparse sequence of images to imitate. The results are in Table 3 . We found that the exploration data collected via curiosity significantly improves the final imitation performance across all methods including the baselines with respect to random exploration. Our baseline GSP model with a forward regularizer instead of consistency loss ends up overfitting to the training layout. In contrast, our forward-consistent GSP model outperforms other methods in generalizing to new map with novel textures. This indicates that the forward consistency is possibly doing more than just regularizing the policy features. Training forward consistency loss in feature space further enhances the generalization even when both pixel and feature space models perform similarly on training environment.

4. RELATED WORK:

Our work is closely related to imitation learning, but we address a different problem statement that gives less supervision and requires generalization across tasks during inference. (Andrychowicz et al., 2017) we draw goals from experience, but our policy optimization has better sample efficiency through supervised learning and dynamics modeling instead of reinforcement learning. Moreover, we work from high-dimensional visual inputs instead of knowledge of the true states and do not make use of a task reward during training. In our setting, all of the expert goals are followed zero-shot since they are only revealed after learning.

5. DISCUSSION:

In this work, we presented a method for imitating expert demonstrations from visual observations alone. In contrast to most work in imitation learning, we never require access to expert actions. The key idea is to learn a GSP using data collected by self-supervised exploration. However, this limits the quality of the learned GSP as per the exploration data. For instance, we deploy random exploration on our real-world navigation robot, which means that it would almost never follow trajectories that go between rooms. Consequently, the learned GSP is unable to navigate towards a goal image taken in another room without requiring intermediate sub-goals. Pathak et al. (2017) show that the agent learns to move along corridors and transition between rooms purely driven by curiosity in Viz-Doom. Training GSP on such a structured data could equip the agent with more interesting search behaviors, e.g., going across rooms to find a goal. In general, using better methods of exploration for training the GSP could be a fruitful direction toward generalizing zeroshot imitation.

One limitation of our approach is that we require first-person view demonstrations. Extension to third-person demonstrations (Liu et al., 2018; Stadie et al., 2017) would make the method applicable in more general scenarios. Another limitation is that, in the current framework, it is implicitly assumed that the statistics of visual observations when the expert demonstrates the task and the agent follows it are similar. For e.g., when the expert performs a demonstration in one setting, say in daylight and the agent needs to imitate say in the evening, the change in the lighting conditions might result in worse performance. Making the GSP robust to such nuisance changes or other changes in environment by domain adaptation would be necessary to scale the method to practical problems. Another thing to note is that, in the current framework, we do not learn from expert demonstrations, but simply imitate them. It would be interesting to investigate ways for an agent to learn from the expert to bias its exploration to more useful parts of the environment.

While we used a sequence of images to provide a demonstration, our work makes no image-specific assumptions and can be extended to using formal language for communicating goals. For instance, after training the GSP, instead of transforming an image into features φ as described in section 2.2, one could possibly learn a mapping to transform language instructions into this feature space.

:

https://pathak22.github.io/zeroshot-imitation/

Formula for computing median confidence intervals: http://www.ucl.ac.uk/ich/ short-courses-events/about-stats-courses/stats-rm/Chapter_8_Content/ confidence_interval_single_median

Figures and Tables:

7 d 4 1 a s 1 G k U Y Y T O I V z 8 O A S m n A L L f C B Q Q b P 8 A p v T u 6 8 O O / O x 6 K 1 5 B Q z x / A H z u c P a L y R a g = = < / l a t e x i t > (d) Forward-consistent GSP (ours) < l a t e x i t s h a 1 _ b a s e 6 4 = " A v M 0 k 9 A 7 Q n q k f A C e B S / 9 I O K X x 8 o = " > A A A C J 3 i c b V B N S w M x E M 3 6 W d e v q k c v w S K 0 B 8 u u C O p J Q V C P F a 0 W u q V k s 9 M 2 N J s s S V Y p S 3 + O F / + K F x E V P f p P T D 8 E t Q 4 M P N6 b x 8 y 8 M O F M G 8 / 7 c K a m Z 2 b n 5 n M L 7 u L S 8 s p q

Figure 1: The goal-conditioned skill policy (GSP) takes as input the current and goal observations and outputs an action sequence that would lead to that goal. We compare the performance of the following GSP models: (a) Simple inverse model; (b) Mutli-step GSP with previous action history; (c) Mutli-step GSP with previous action history and a forward model as regularizer, but no forward consistency; (d) Mutli-step GSP with forward consistency loss proposed in this work.

the rope is fixed.

Figure 2: Qualitative visualization of results for rope manipulation task using Baxter robot. (a) Our robotics system setup. (b) The sequence of human demonstration images provided by the human during inference for the task of knot-tying (top row), and the sequences of observation states reached by the robot while imitating the given demonstration (bottom rows). (c) The sequence of human demonstration images and the ones reached by the robot for the task of manipulating rope into 'S' shape. Our agent is able to successfully imitate the demonstration.

r b 9 g 5 9 + 4 P A o F T 3 V y z r 2 5 5 5 4 g l s K i 5 3 0 7 m a X l l d W 1 7 H p u Y 3 N r e 8 f d 3 b u 3 O j E c q l x L b e o B s y C F g i o K l F C P D b A o k

v J I 3 5 8 l 5 c d 6 d j + l o x p n t 7 J M / c D 5 / A J G G m J Q = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c B f 7 Q n 6 Y b k K

r b 9 g 5 9 + 4 P A o F T 3 V y z r 2 5 5 5 4 g l s K i 5 3 0 7 m a X l l d W 1 7 H p u Y 3 N r e 8 f d 3 b u 3 O j E c q l x L b e o B s y C F g i o K l F C P D b A o k

Figure 3: GSP trained using forward consistency loss significantly outperforms the baselines at the task of (a) manipulating rope into 'S' shape as measured by TPS-RPM error and (b) knot-tying where we report success rate with bootstrap standard deviation.

Figure 4: Visualization of the TurtleBot trajectory to reach a goal image (right) from the initial image (top-left). Since the initial and goal image have no overlap, the robot first explores the environment by turning in place. Once it detects overlap between its current image and goal image (i.e. step 42 onward), it moves towards the goal. Note that we did not explicitly train the robot to explore and such exploratory behavior naturally emerged from the self-supervised learning.

Run Id-2 Run Id-3 Run Id-4 Run Id-5 Run Id-6 Run Id-7 Run Id-8 Num Success

Table content: <table><tr><td colspan="2">Model Name Run Id-1 Random Search Fail Inverse Model [Nair et. al. 2017] Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>0 0</td></tr><tr><td colspan="2">GSP-NoPrevAction-NoFwdConst 39 steps GSP-NoFwdConst 22 steps</td><td>34 steps 22 steps</td><td>Fail 39 steps</td><td>Fail 48 steps</td><td>Fail Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>Fail Fail</td><td>2 4</td></tr><tr><td>GSP (Ours)</td><td colspan="4">119 steps 66 steps 144 steps 67 steps</td><td>51 steps</td><td>Fail</td><td>100 steps</td><td>Fail</td><td>6</td></tr></table>

Imitation LearningThe two main threads of imitation learning are behavioral cloning(Argall et al., 2009;Pomerleau, 1989), which directly supervises the mapping of states to actions, and inverse reinforcement learning(Abbeel & Ng, 2004;Ho & Ermon, 2016;Levine et al., 2016;Ng & Russell, 2000;Ziebart et al., 2008), which recovers a reward function that makes the demonstration optimal (or nearly optimal). Inverse RL is most commonly achieved with state-actions, and is difficult to extend to fitting the reward to observations alone, though in principle state occupancy could be sufficient. Recent work in imitation learning(Duan et al., 2017;Finn et al., 2017;Gupta et al., 2017) can generalize to novel goals, but require a wealth of demonstrations comprised of expert state-actions for learning. Our approach does not require expert actions at all.Visual DemonstrationThe common scenario in LfD is to assume full knowledge of expert states and actions during demonstrations, but several papers have focused on relaxing this supervision to visual observations alone.Nair et al. (2017) observe a sequence of images from the expert demonstration for performing rope manipulations.Sermanet et al. (2017; 2018)  imitate humans with robots by self-supervised learning but require expert supervision at training time. Third person imitation learning(Stadie et al., 2017) and the concurrent work of imitation-from-observation(Liu et al., 2018) learn to translate expert observations into agent observations such that they can do policy optimization to minimize the distance between the agent trajectory and the translated demonstration, but they require demonstrations for learning. Visual servoing is a standard problem in robotics(Koichi & Tom, 1993) that seeks to take actions that align the agent's observation with a target configuration of carefully-designed visual features(Wilson et al., 1996;Yoshimi & Allen, 1994) or raw pixel intensities(Caron et al., 2013). Classical methods rely on fixed features or policies, but more recently end-to-end learning has improved results(Lampe & Riedmiller, 2013;Lee et al., 2017). Forward/Inverse Dynamics and Consistency Numerous prior works, such as Ebert et al. (2017); Oh et al. (2015); Watter et al. (2015), have learned forward dynamics model for planning actions. The works of Agrawal et al. (2016); Jordan & Rumelhart (1992); Pathak et al. (2017); Wolpert et al. (1995) jointly learn forward and inverse dynamics model but do not optimize for consistency between the forward and inverse dynamics. We empirically show that learning models by our forward consistency loss significantly improves task performance. Enforcing consistency as a meta-supervision has also been successful in finding visual correspondences(Zhou et al., 2016) or unpaired image translations(Zhu et al., 2017).Goal Conditioning By parameterizing the value or policy function with a goal, an agent can learn and do multiple tasks. The idea of learning goal-conditioned policies has been explored in(Agrawal et al., 2016;Andrychowicz et al., 2017;Nair et al., 2017;Schaul et al., 2015). Similarly to hindsight experience replay

Table content: <table/>

Peer review (iclr2018):
 One of the main problems with imitation learning in general is the expense of expert demonstration. The authors here propose a method for sidestepping this issue by using the random exploration of an agent to learn generalizable skills which can then be applied without any specific pretraining on any new task. The proposed method has at its core a method for learning a parametric skill function (PSF) that takes as input a description of the initial state, goal state, parameters of the skill and outputs a sequence of actions (could be of varying length) which take the agent from initial state to goal state. The skill function uses a RNN as function approximator and minimizes the sum of two losses i.e. the state mismatch loss over the trajectory (using an explicitly learnt forward model) and the action mismatch loss (using a model-free action prediction module) . This is hard to do in practice due to jointly learning both the forward model as well as the state mismatches. So first they are separately learnt and then fine-tuned together. In order to decide when to stop, an independent goal detector is trained which was found to be better than adding a *goal-reached* action to the PSF. Experiments on two domains are presented. 1. Visual navigation where images of start and goal states are given as input. 2. Robotic knot-tying with a loose rope where visual input of the initial and final rope states are given as input. Comments: - In the visual navigation task no numbers are presented on the comparison to slam-based techniques used as baselines although it is mentioned that it will be revisited. - In the rope knot-tying task no slam-based or other classical baselines are mentioned. - My main concern is that I am really trying to place this paper with respect to doing reinforcement learning first (either in simulation or in the real world itself, on-policy or off-policy) and then just using the learnt policy on test tasks. Or in other words why should we call this zero-shot imitation instead of simply reinforcement learnt policy being learnt and then used. The nice part of doing RL is that it provides ways of actively controlling the exploration. See this pretty relevant paper which attempts the same task and also claims to have the target state generalization ability. Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning by Zhu et al. I am genuinely curious and would love the authors* comments on this. It should help make it clearer in the paper as well. Update: After evaluating the response from the authors and ensuing discussion as well as the other reviews and their corresponding discussion, I am revising my rating for this paper up. This will be an interesting paper to have at the conference and will spur more ideas and follow-on work.