RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Meta-world
Experiments: In order to run experiments and baseline of MetaWorld [ 81 ] , we used the training routines and functions offered in the python library garage [ 82 ] . We ran two different types of experiments:

ML1 : In this setting, we restrict the task distribution to a single type of robotic manipulation while varying the goal location. The meta-training “tasks” corresponds to 50 random initial agent and goal positions, and meta-testing to 50 heldout positions. We tested our model on the reach-v2 and push-v2 tasks.

ML10 : This set-up tests generalization to new manipulation tasks, the benchmark provides 10 training tasks and holds out 5 meta-testing tasks.

Training details: We used PPO [ 86 ] for training our model, with clip range r=0.2 and 10 inner optimization steps maximum, for its good empirical performance and in order to compare with other methods with recurrent computation scheme. We kept the experimental settings from the original paper with N=10 episodes of 500 time-steps and sampled proximal gradient update in a meta-batch size of 25 trajectories. To estimate value function, we both tested a feedforward neural network trained at each iteration of the meta-training or an additional MetODS network, which resulted in similar training results.
Figure 2 : A layer of MetODS updates its weights by recursive applications of read and write operations based on neural activations \bm{v}^{(s)} and synaptic traces \bm{W}^{(s)} .; A layer of MetODS updates its weights by recursive applications of read and write operations based on neural activations \bm{v}^{(s)} and synaptic traces \bm{W}^{(s)} .
Figure 3 : a-b) Schemas of the Harlow and Mujoco Ant-directional locomotion task. c-d) Evolution of accumulated reward over training. In the Harlow task, we conduct an ablation study by either reducing the number of recursive iterations (S=1) or removing the trainable plasticity weights \bm{\alpha} resulting in sub-optimal policy. In Ant-dir we compare our agent training profile against MAML and RL 2 . e) We can interpret the learned policy in terms of a Hopfield energy adapting with experience. We show horizontally two reward profiles of different episodes and the energy E_{\bm{W}_{t}}(v_{1},v_{2})=-v^{T}_{1}\bm{W}_{t}v_{2} along two principal components of the vector trajectory \bm{v_{t}} . In the first episode, the error in the first presentation (red square) transforms the energy landscape which changes the agent policy, while on the other episode, the model belief does not change over time. Note the two modes for every energy map, which allows the model to handle the potential position permutation of the presented values. f) Average rewards per timestep during a single episode of the Ant-dir task.; a-b) Schemas of the Harlow and Mujoco Ant-directional locomotion task. c-d) Evolution of accumulated reward over training. In the Harlow task, we conduct an ablation study by either reducing the number of recursive iterations (S=1) or removing the trainable plasticity weights \bm{\alpha} resulting in sub-optimal policy. In Ant-dir we compare our agent training profile against MAML and RL 2 . e) We can interpret the learned policy in terms of a Hopfield energy adapting with experience. We show horizontally two reward profiles of different episodes and the energy E_{\bm{W}_{t}}(v_{1},v_{2})=-v^{T}_{1}\bm{W}_{t}v_{2} along two principal components of the vector trajectory \bm{v_{t}} . In the first episode, the error in the first presentation (red square) transforms the energy landscape which changes the agent policy, while on the other episode, the model belief does not change over time. Note the two modes for every energy map, which allows the model to handle the potential position permutation of the presented values. f) Average rewards per timestep during a single episode of the Ant-dir task.
Figure 4 : a) Examples of generated maze configurations. The mazes consist in 8 × 8 pix. areas, with walls and obstacles randomly placed according to a variation of Prim’s algorithm [ 80 ] with target location randomly selected for the entire duration of the episode (star). The agent receptive field is highlighted in red. b) Comparisons with MAML and RL 2 and effect of element-wise plasticity \bm{\alpha} . c) Variations of MetODS writing mechanism as well as depth S of recursivity yield different performances (see. S.I for further details); a) Examples of generated maze configurations. The mazes consist in 8 × 8 pix. areas, with walls and obstacles randomly placed according to a variation of Prim’s algorithm [ 80 ] with target location randomly selected for the entire duration of the episode (star). The agent receptive field is highlighted in red. b) Comparisons with MAML and RL 2 and effect of element-wise plasticity \bm{\alpha} . c) Variations of MetODS writing mechanism as well as depth S of recursivity yield different performances (see. S.I for further details)
Figure 5 : Performance of Meta-RL models tested at convergence ( 1e^{7} env. steps). MetODS better explores the maze as measured by the average number of steps before 1st reward and the success rate in finding the reward at least once. It then better exploits the maze as per the accumulated reward. (* We assign 100 to episodes with no reward encounter.); Performance of Meta-RL models tested at convergence ( 1e^{7} env. steps). MetODS better explores the maze as measured by the average number of steps before 1st reward and the success rate in finding the reward at least once. It then better exploits the maze as per the accumulated reward. (* We assign 100 to episodes with no reward encounter.)
Figure 6 : Left Meta-training results for MetaWorld benchmarks. Subplots show tasks success rate over training timesteps. Average meta-test results for MetODS is shown in dotted line. Right Cumulative reward of the Ant and Cheetah directional locomotion task. For each condition, results are normalized against the best performing policy; Left Meta-training results for MetaWorld benchmarks. Subplots show tasks success rate over training timesteps. Average meta-test results for MetODS is shown in dotted line. Right Cumulative reward of the Ant and Cheetah directional locomotion task. For each condition, results are normalized against the best performing policy
Figure 7 : Harlow task: a) An episode of the Harlow task. From left to right: Actions, visual field of the agent (blue squares:fixation target, red squares: values), Rewards (red;negative,light blue: fixation reward, dark blue:value reward) and sum of absolute synaptic variation \|\sum\Delta\bm{W}\| per neuron. b) Distribution of weights before and after adaptation over 5 presentations. c) Three instances of the dynamic weights \bm{W}_{t} after adaptation that solved the Harlow task. Every synaptic configuration presents differences but perform optimally.; Harlow task: a) An episode of the Harlow task. From left to right: Actions, visual field of the agent (blue squares:fixation target, red squares: values), Rewards (red;negative,light blue: fixation reward, dark blue:value reward) and sum of absolute synaptic variation \|\sum\Delta\bm{W}\| per neuron. b) Distribution of weights before and after adaptation over 5 presentations. c) Three instances of the dynamic weights \bm{W}_{t} after adaptation that solved the Harlow task. Every synaptic configuration presents differences but perform optimally.
Figure 8 : Reward profiles of MAML, RL 2 and MetODS over 5 consecutive episodes with randomly varying rewarded direction. In Cheetah , MetODS slightly overperforms RL 2 and in Ant , MetODS is the only approach able to reach performance comparable to first episode in subsequent episodes.; Reward profiles of MAML, RL 2 and MetODS over 5 consecutive episodes with randomly varying rewarded direction. In Cheetah , MetODS slightly overperforms RL 2 and in Ant , MetODS is the only approach able to reach performance comparable to first episode in subsequent episodes.
Figure 9 : Meta-training rewards curves of the maze experiment presented in section 5.2; Meta-training rewards curves of the maze experiment presented in section 5.2
Figure 10 : Heatmap of activation rate of three neurons of a trained model across 10 different maze maps normalized by agent passage. Activation patterns show selectivity to particular positions within an episode and strong remapping across maze configurations.; Heatmap of activation rate of three neurons of a trained model across 10 different maze maps normalized by agent passage. Activation patterns show selectivity to particular positions within an episode and strong remapping across maze configurations.