On the Importance of Distraction-Robust Representations for Robot Learning
Authors: Andy Wang
Antoine Cully
Venue:  (2021)
Keywords: Unsupervised Representation Learning
Robot Control
Quality-Diversity

Abstract: Representation Learning methods can allow the application of Reinforcement Learning algorithms when a high dimensionality in a robot's perceptions would otherwise prove prohibitive. Consequently, unsupervised Representation Learning components often feature in robot control algorithms that assume high-dimensional camera images as the principal source of information.
In their design and performance, these algorithms often benefit from the controlled nature of the simulation or laboratory conditions they are evaluated in. However, these settings fail to acknowledge the stochasticity of most real-world environments.
In this work, we introduce the concept of Distraction-Robust Representation Learning. We argue that environment noise and other distractions require learned representations to encode the robot's expected perceptions rather than the observed ones. Our experimental evaluations demonstrate that representations learned with a traditional dimensionality reduction algorithm are strongly susceptible to distractions in a robot's environment.
We propose an Encoder-Decoder architecture that produces representations that allow the learning outcomes of robot control tasks to remain unaffected by these distractions.

Under review as a conference paper at ICLR 2021
ON THE IMPORTANCE OF DISTRACTION-ROBUST REPRESENTATIONS FOR ROBOT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Representation Learning methods can allow the application of Reinforcement Learning algorithms when high dimensionality in robot's perceptions would otherwise prove prohibitive. Consequently, unsupervised Representation Learn- ing components often feature in robot control algorithms that assume high- dimensional camera images as the principal source of information. In their design and performance, these algorithms often benefit from the controlled nature of the simulation or the laboratory conditions they are evaluated in. However, these set- tings fail to acknowledge the stochasticity of most real-world environments. In this work, we introduce the concept of Instraction-Robuse Representation Learn- ing. We argue that euvironment noise and other distractions require learned rep- resentations to encode the robot's expected perceptions rather than the observed ones. Our experimental evaluations demonstrate that representations learned with traditional dimensionality reduction algorithm are strongly susceptible to dis- architecture
tractions in a robot's environment. We propose an that produces representations that allow the learning outcomes of robot control tasks to remain unaffected by these distractions.
INTRODUCTION
Representation Learning techniques form an integral part in many Reinforcement Learning (RL) robot control applications (Lesort al. 2018). Utilising low-dimensional representations can allow for a faster and more efficient learning of tasks than when using high-dimensional sensor informa- tion (Munk et al. (016)) This particularly useful in vision-based learning when high-dimensional images of the robot's 's environment are the principal source of information available to the learning algorithm (Zhu et al. 2020). Most commonly, representations are learned by applying dimensional- ity reduction techniques such Autoencoders (AEs) (Hinton & Salakhutdinov (2066) Variational Autoencoders (VAEs) (Kingma& Welling 2014) to the robot's 's sensory data (Lange et al. (0022) Zhu Encoder E. and Decoder 7). The Encoder attempts to condense all available information in the input data æ: into a latent repre- sentation from which reconstruction of the inputs ((F(x)) is generated by the Decoder. When the dimensionality of the representation is smaller than that of the input data, some information is lost when creating the representations. An AE typically trained to shrink the magnitude of this information loss by minimising reconstruction error. This error is commonly given by the squared norm of differences,
al. 2020). Generally, an AE consists of two Neural Networks,
LAE
x D(E(x)))
However, the optimisation of the reconstruction error in Eq. [ does not necessarily result in the gen- eration of representations that are optimal for the use in robot learning algorithms. For example, an accurate reconstruction of the decorative patterns on dinner plate is less important than the plate's dimensions to a robot learning to place it into a cabinet. can therefore be desirable to control which aspects of the information contained in the inputs are most critical to be preserved in the represen- tations. For instance, Pathak et al. (201)) design a Neural Network that learns representations from visual inputs by using them predict the action taken by the RL agent in its state transition. By ask- ing the network to predict the action, the authors eliminate the requirement for the representations to retain any state information that is unrelated the agent's 's behaviour.

---

Under review as a conference paper at ICLR 2021
A focus on the learned representations' preservation of task-relevant information becomes even more crucial in the presence of distracting inffuences (DIs) in the environment. These DIs can materialise in the presence of additional environment objects which exhibit dynamies that are not only uncorrelated with the robot's behaviour but additionally misleading. For instance, a robot that is learning move objects different positions in a room can find the observation of moving autonomous vacuum cleaner misleading. Alternatively, DIs can impact the dynamics of existing objects in the room. For instance, after the robot has moved a box certain position in the room, further movements of the box due to external forces can be distracting to the robot's s learning process.
In this paper, we introduce the concept of Distraction Robuust Representation Learning. We investi- gate the learning outcomes of robot control tasks when DIs are present in the environment. We show that in the presence of DIs, representations learned exclusively from environment observations can mislead the robot's perceptions of its control over the environment. This finding demonstrates that Distraction-Ron Robust Representation Learning needs to be afforded increased attention. In particular, works in the strand of research that aim to make RL algorithms more applicable to real-world sce- narios largely concentrate on improving algorithm attributes such as the data efficiency (Zhu et al, 2020). However, few works acknowledge the challenges posed by the inherently stochastic nature of real-world environments and the presence of DIs (Forestier et al., 2017). Furthermore, we introduce Robot Action Encoder-Decoder architecture (RAED) which successfully produces representations that are robust to DIs in the environment. RAED follows the simple but effective approach of us- ing only the values that parameterise the robot's actions as the input the Encoder. Such set of parameters defines a robot controller for instance. The representations produced by the Encoder are used by RAED's Decoder to generate predictions of the environment observations. RAED's design allows for static environment elements to be learned by the Decoder while concentrating the infor- mation in the representations on the observable consequences of the robot's behaviour. Moreover, when environment observations are distorted by the presence of DIs, RAED produces representa- tions that capture the expected consequences of the robot's environment interactions. This is not the case when training representation learning methods such as AEs to reconstruct the full content of the robot's visual perceptions. We can therefore draw some parallels between RAED's s design and the concept of forward model (Jordan & Rumelhart 1992). Given the simplicity of the approach, we expect the applicability of RAED to generalise to various different learning algorithms.
2 RELATED WORK
Several works have investigated mechanisms to preserve only task-relevant information in learned representations. Pathak al. (017) propose a Neural Network architecture that learns represen- tations of visual inputs by predicting the action taken by the RL agent in its state transition. This design allows the representations
dedicate their information capacity to the observable conse- quences of the robot's 's actions. Finn et al. (2016) propose spatial AE to learn representations that aim to preserve only the configuration of objects in the environment rather than all aspects of the information contained in the camera images. However, in both approaches, the representations learned from visual inputs which will be distorted if DIs are present in the environment. Without an explicit correction mechanism, these representation learning techniques therefore remain suscepti- ble to distractions. The concept of affordance learning formulates a similar goal in discovering the consequences of the robot's actions on its environment (Cakmak et al., 2007 Şahin et al. (007). However, the works in this strand of research rarely consider the problem of DIs in the environment. Instead, they mainly concentrate on the robot's is ability infer how an object in the environment would behave in response to its actions when no prior interaction experience with that particular object is available (Dehban et al., 2016: Mar et al. 2015).
A work that uses learned representations and investigates robot interactions in the presence of DIs presented in the Intrinsically Motivated Goal Exploration Processes (IMGEP, Laversanne-Finot et al. (2018)). IMGEP aims to enable robots to explore the possible interactions with various tools in an environment that also features distractor objects. These objects either cannot be interacted with or move independently of the robot. The authors show that variant of their proposed algorithm re- mains unaffected by the presence of these distractors. This robustness is demonstrated by the robot's s lack of interaction with the distractor objects. However, the DIs we consider in this paper pose an arguably larger challenge for two main reasons. First, we evaluate distractor objects which exhibit dynamics that not only independent of the robot's behaviour but also misleading the robot's
2

---

Under review as a conference paper at ICLR 2021
perceptions of its interactions with the actual objects of interest. For instance, a distractor object can collide with one of the objects manipulated by the robot. Second, we consider the case of mislead- ing external DIs the interesting objects themselves. In this case, the object interaction cannot be avoided. Furthermore, in IMGEP the distractor robustness not learned by the VAE that generates the representations. Instead, the algorithm identifies the components in the representations that cor- respond to the distractors during the robot's interaction phase. Indeed, the learned representations remain static in this phase as the VAE is pre-trained on manually generated dataset that covers all possible environment observations by spawning the various environment objects, including the distractors, in different positions.
Alternative exploration- based approaches have been proposed recently (Forestier et al. [011 Pathak Pathak
jt al. 2017: Eysenbach et al. 22199; [haamma al., 2019). particularly interesting framework presented in AURORA (Cully (2019), which we use in our experimental evaluations. Instead of decoupling the representation learning phase from the interaction phase as done in IMGEP, AU- RORA proposes joint approach to discover diverse set of behaviours of any robot in any given environment without the need to specify a task objective or reward function. More precisely, AU- RORA applies dimensionality reduction algorithm such as an AE to the robot's sensory data that are collected during the execution of the discovered controllers. The learned latent representation associated to controller's collected sensory data defines the Behavioural Descriptor (BD) for this controller. By measuring the distance between different controllers' BDs, the novelty of a given behaviour can be determined. The BD can therefore be used to build a collection of controllers that each exhibit different behaviours. AURORA uses this collection to further explore the space of possible behaviours via stochastic mutations, and continuously adds additional novel controllers to the collection. Periodically, the AE is trained using the sensory data that are extracted from all controllers in the collection.
The AURORA framework is well-suited to judge the impact of DIs on the learned representations. Indeed, AURORA's decision to accept newly generated controllers into the collection is exclusively based on the distance between their BDs. We can therefore analyse the influence of DIs using metrics that are formulated directly in the latent representation space. These metries are more sensitive to the impact of DIs than measures of the overall learning outcomes. This is because the learning outcomes are typically also influenced by components of the learning algorithm other than the representations. Furthermore, AURORA'S training of the dimensionality reduction technique provides a particularly challenging setting for RAED. The training dataset composition is continually evolving as the data collected online. Additionally, AURORA rejects controllers that generate behaviours very similar to that of existing controllers. This complexifies the task of learning the expected consequences of the robot's actions when the collected environment observations are distorted by DIs.
METHOD
In this work, we use a single vector a to parameterise the set of actions taken by the robot from the beginning until the end of the simulation. This vector of the robot's actions can store the parameter values of a controller for instance. We use a vector æ to parameterise the single environment ob- servation that is returned by each simulation. This vector can store the pixel values of image for example.
RAED consists of Encoder- structure that uses a and æ to produce a representation = that is distraction-robust. More precisely, the latent representation z is generated from RAED's Encoder network E which takes the robot's actions a as its inputs. This conditioning of the representations on the robot's 's actions is the crucial element in RAED's 's design. The Decoder uses the representations to produce a prediction of the environment observation D D(E(a)). The actual environment observation is used the prediction target in the training of RAED. This yields the prediction error given in Eq. 2 Note, the difference in the Encoder input compared to an AE's reconstruction error in Eq. [
C C=
Depending the specifics of the robot design and the task, set of controllers with different pa- rameters can yield the same observed behaviour. For instance, when stecring a robot arm with redundancies in its joints, different controllers can achieve the same position in gripper connected
3

---

Under review as a conference paper at ICLR 2021
to the arm. For such a set of controllers, it is possible that the differences in their parameters lead to the generation of different representations when using RAED, even if the observation predictions exactly the same. This is less likely to occur in a dimensionality reduction technique that takes the visual perceptions as its inputs. For these methods, the principal sources of variation in the rep- resentations are the differences in the observations rather than the differences in the robot's actions. To reduce RAED's freedom to disperse the representations of similar observed behaviours, we in- vestigate the addition of penalty term P. The overall training loss then given by LRAED C-
One possible instantiation of P is the Kullback-Leibler divergence (KL) term used in a VAE (Kingma & Welling 2014). This KL term implicitly imposes a penalty on the magnitude of the means produced by the VAE'S Encoder network. The consequence is crowding of the representa- tions into a smaller latent subspace. This can have the desired effect of reducing RAED's ' freedom to produce dissimilar representations for similar observed behaviours. However, the practical im- plementation of the KI. term also imposes a penalty on any Encoder variance magnitudes different than 1. Large Encoder variances result in an increased dispersion of representations when these are sampled from the Encoder. The presence of these variances can therefore be interpreted as a fur- ther source of stochasticity when learning representations. Our experimental results in Section 4.4 eeeerimeeee
demonstrate that the application of the KL term benefits from leaving the Encoder variance outputs unused. The derived penalty can be summarised in simple linear term given by Pinear ||
pevndor
Sanms
RAED
conmolerrs
Crass-Ovaz
Musetion
Nira controllers
Robot sotione
RD distance cissest diion d
colection
Betaviounal
Daraakon
Discard soludor
Figure 1: AURORA algorithm with a modular integration of RAED.
In the AURORA framework, RAED can be used as a modular representation learning component to produce the BDs for the discovered controllers. We present an illustration of AURORA in Fig. [ and a pseudo-code in Algorithm | in Appendix B. AURORA largely follows standard Quality- Diversity algorithm (Cully & Demiris 2018a). the start of each iteration, a predefined number of controllers is selected from the collection. This group of controllers forms the set of parent con- trollers. Each pair of parents undergoes cross-over and mutation operations. Cross-over operations produce controllers that bear similarities to both parents. Mutation operations introduce some stochastic changes to these new controllers. If the collection is empty at the start of the iteration, we instead generate new controllers by randomly sampling in the controller space.
Each controller is executed in the simulated environment and the associated observations are cap- tured. These observations are used to produce the BD of each controller. When using an AE in AURORA, each BD is the latent representation generated using the environment observation æ as input. In RAED, the robot's actions a are the inputs instead. AURORA, therefore, stores for each controller a vector of robot action parameters, an environment observation and a BD.
In the decision to accept a given new controller into the collection, AURORA considers the distance between its BD to the descriptor of the nearest controller contained in the collection. If this distance exceeds a minimum distance threshold d, the controller is deemed sufficiently novel to be added to the collection (Cully & Demiris 2018b). Otherwise, it is discarded. d is initialised at the start of the algorithm using the BDs of the initial set of controllers contained in the collection. AURORA then updates the value of d after each iteration to steer the size of the collection towards the target size.
Every N iterations, AURORA extracts the environment observations and robot actions from all con- trollers contained in the collection to train the AE or RAED respectively. Once the training of the network concludes, all controllers are removed from the collection and assigned new BDs. This set of controllers is finally re-added to the collection using the acceptance mechanism described in the previous paragraph. This concludes one iteration of the AURORA algorithm. The hyperparameters used in AURORA given in Appendix B.1
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
In our experimental evaluations, we study the impact of DIs on RAED and on AE that uses visual perceptions as its inputs. The learned representations are two-dimensional and used by AURORA to generate a diverse set of robot behaviours. We consider an experiment setup of a planar robot arm with four degrees of freedom mounted the center table. The arm interacts with environment objects in two different scenarios: 1) an object arrangement scenario, and 2) an air-hockey scenario. Each scenario evaluates a different type of which we detail in the following paragraphs. Fig. 2 presents the simulated environment and an illustrative environment observation for each scenario. The configuration and hyperparameters of the simulations are given in Appendix C
:
Figure 2: Environment simulations and observations. From left to right: (a) Object arrangement scenario initial simulation state without any distracting influences (DIs). (b) Air-hockey scenario
initial simulation state without DIs. (c) Robot arm pushing an object the environment. (d) Object arrangement environment observation with DIs. (e) Air-hockey environment observation with DIs.
4.1 SCENARIO 1: OBJECT ARRANGEMENT
The object arrangement scenario features two objects positioned the table shown in Fig. P The robot arm can push these objects as shown in Fig. 2] but they do not move easily due to their large weight. Each controller is two-staged and consists of parameter values. The first 4 parameters dictate the angles in the 4 joints of the arm to be reached by the end of the first half of the simulation. The latter dictate the angles be reached by the end of the second half. The robot's environment interactions are simulated for 10 seconds. The environment observation consists of 40 x 40 image that captures the objects' positions the end of the simulation. The robot arm itself does not feature in the observations. The DIs materialise in external force that pushes one of the two objects meddling
in random direction at the start of the simulation. This DI simulates the presence human while our household robot is learning to arrange objects in the environment. We simulate several DI frequency levels which we refer to as the environment stochasticity. At 0.5 environment stochasticity, there is 50% probability that the external force is applied in any given simulation. The object to be pushed is selected randomly. Fig. 21 presents an environment observation after a DI has shifted the lower object from its starting position.
4.2 SCENARIO 2: AIR-HOCKEY
This scenario features an air-hockey puck that spawns in the same starting location at the beginning of each simulation shown in 2p. The robot arm strike the puck as illustrated in Fig. D The puck glides on the table surface and can bounce off the walls. controller now consists only of 4 parameters which dictate the angles in the joints of the arm to be reached by the end of the simulation. The environment observation consists of a 40 40 long-exposure image that traces out the trajectory of the puck. The robot arm does not feature in the observations. DIs are simulated by the addition of a second puck. This puck appears in random position at the start of the simulation and moves in a random direction at a random velocity. This "random" puck is also captured in the environment observation. At 0.5 environment stochasticity, the random puck appears in any given simulation with 50% probability. This DI simulates the presence of an additional player with their puck while the robot is perfecting its air- air-hockey game. Fig. 2 shows environment observation of the random puck's trajectory in the bottom right of the image and the non non-random puck's trajectory to its left.
5

---

Under review as a conference paper at ICLR 2021
4.3 PERFORMANCE METRICS
We calculate two performance metrics: 1) the behavioural diversity, and 2) the proportion of no- move controllers. We perform 5 repetitions of each evaluation obtain robustness in the results.
The behavioural diversity common way to evaluate the quality of AURORA'S learning out- come (Cully) 2019). measures the diversity of the behaviours that are generated by all controllers in the collection. We quantify this diversity as the variance in the positions of the objects of interest. In the arrangement scenario, we extract both environment objects' positions the end of the sim- ulation. We calculate the variance in each object's 's position across all controllers in the collection and average these two scores obtain our metric. In the air-hockey scenario, we extract for each controller the non-random puck's s trajectory coordinates. For each trajectory element, we calculate average variances in x and y-coordinates across controllers the collection to obtain score. The overall diversity in the behaviours is measured the average of the scores over all trajectory elements. In both scenarios, a larger variance score therefore shows an improved learning outcome. If DIs affect the learned representations, controllers Very similar controllers can then be accepted repeatedly into the collection. A susceptibility to DIs would therefore show in decreasing variance scores as the environment stochasticity increases.
be misdescribed by their BDs.
In both scenarios, some arm movements do not produce any change in the objects' positions. Since the robot itself is not captured in the environment observation, such "no-move" controllers produce the same observation when no DIs are present in the environment. Their BDs are therefore identical and AURORA allows at most one such no-move controller to be present in the collection at any given time. However, controllers that do not move any objects in distraction-free free environment do not appear as such whenever DIs are present. If the DIs distort these controllers' BDs, AURORA considers them different from other no-move controllers. Consequently, a presence of DIs can fool AURORA into accepting multiple no-move controllers into the collection. We therefore use the proportion of no-move controllers in the collection the DIs on the learned representations. If this metric increases in the environment stochasticity, we conclude that the generation of representations is susceptible to DIs in the environment.
second metric to quantify the impact of
4.4 EXPERIMENTAL RESULTS
Variance Positions of Objects of Interest (Larger Better) Object Arrangement Task
Position of Objects at End of Simulation
Air-Hockeyy Task
AE
0.50
0.45
0.9
0.35
0.30
AE
RAED
0.7
MAED-LPENALTY RAED INPENNLTTY
0.5
Object 2
Object
0.20
0.6 0.8
0.2
5.0
Envirarment Sttchanbicily
Figure 3: The variance in the environment ob- jects' positions. Left (a): Object arrangement task. Right: (b) Air-hockey task. The lines rep- resent the median scores while the shaded areas extend the first and third quartile of the data distribution obtained over 5 repetitions.
Figure 4: Positions of the two objects in the rangement task after the execution of all con- trollers in collections generated at 100% vironment stochasticity. AURORA achieves larger exploration in the object configurations when using RAED.
Fig. shows the behavioural diversity achieved in both scenarios when using the AE, RAED and two additional variants of RAED in which we add the KL term and the linear penalty term Pinear respectivell. Note, that the AE and RAED's variants have larger network capacity in the air-hockey scenario than in the arrangement task. This is to account for the more complex observation type of a long-exposure image in the hir-hockey task, compared to the snapshot of the environment objects' positions the arrangement scenario (see Fig. 2 In both scenarios. the use
from Section 3
of RAED's learned representations leads to stable variance scores in Figs. 3 and 3p across all

---

Under review as a conference paper at ICLR 2021
evaluated DI frequency levels. This stability demonstrates that RAED's representations robust to distractions and encode the expeeted consequences of the robot's 's actions rather than the observed ones. Conversely, the use of an AE leads to decreasing trends in the variance scores in Figs. B and 33.. These negative trends indicate that the robot's perceptions of its behaviour are distorted in the AE's learned representations due to the DIs in the environment. As a result of this distortion, controllers that produce very similar object configurations in the arrangement task, and very similar puck trajectories in the hockey task, are assigned BDs that make them appear different from each other. Consequently, when using the AE, AURORA repeatedly accepts similar controllers into the collection. This susceptibility to distractions leads to decreasing variance scores for the AE as DIs grow more frequent. At 100% environment stochasticity, the use of an AE leads to median variance scores of 0.31 and 0.78 in the arrangement and air-hockey task respectively. These are significantly lower than the scores of 0.55 and 1.08 when using RAED. In both scenarios, the two methods' differences in the learning outcomes can be seen in visualisations of the exploration in the achievable behaviours. At 100% environment stochasticity, the use of RAED leads to wider exploration in the possible object configurations in the arrangement task in Fig. 4 Fig. 5 shows a substantially larger spread in the trajectories of the non-random air-hockee puck when using RAED. Similarly, large differences can be observed between the AE and RAED methods in their network outputs. We provide some illustrative examples in Appendix A
Ar-lbckey Trajectotes Generated Controlleer AP
Proportion Object (Arrengement Task)
fieling
BAFO
100
Both Object Land Object 2
Ron-Rendom Airttockey Pack
100
80
50
RAED-KL PENALTY
RAED LINPENALTY
50
20
02
0.6
0.5
0.5
0.4 0.5 0.5 022 0.4
6.5 88
Figure 5: Puck trajectories generated by controllers in the collection at 100% environment stochasticity. RAED achieves a larger spread in the trajectories.
Figure 6: From left to right: Proportion of controllers failing to move (a) any object of interest in the object arrangement task, (b) both Object and in the object arrangement task, (c) the non con-random puck in the air-hockey task. The lines and shaded areas are obtained as described in Fig. 3
Fig. 6ishows the proportion of controllers in the collection that fail to move the objects of interest in the two scenarios. As explained previously, an increasing trend in this metric indicates the learned representations' sensitivity to the presence of DIs. In the arrangement task, RAED produces stable score across all environment stochasticities in Fig. [p. This affirms the distraction-robustee robustness of RAED's learned representations. In the air- task, there is only one object that the robot needs to learn to control. Compared to the arrangement scenario, this increases the proportion of all controllers that lead to movement in the object of interest. This increases the complexity of the task and leads to a slightly positive trend in Fig. 6 when using RAED. Nevertheless, RAED's learned representations in the generated collections as we saw in Fig. 3p. As expected, the AE's scores are more strongly affected by the DIs in the environment. In the air-hockey task, Fig. C shows rapidly increasing proportion of no-move controllers. Similarly, Fig.lop 6p shows a rising proportion of controllers failing to move both objects in the arrangement task. However, the proportion of controllers failing to move any object in this task when using the AE provides an interesting result. Fig, 6p shows that 0% environment stochasticity, the use of the AE's learned representations leads a minimal proportion of no-move controllers. This is expected, as such a distraction- not provide any challenges to the AE. Surprisingly, however, even as the environment stochasticity grows larger, the use of the AE'S representations persistently leads to low proportions of no-move controllers. This stability appears to indicate that the learned representations are insusceptible to the DIs in the environment. However, the AE variant's sharply decreasing variance scores in Fig. 3 show a strong sensitivity to the environment distractions in this task. The reconciliation of these two contradicting observations the likelihood for any given controller to produce a movement in at least one of the objects is very large for two reasons. First, both objects are placed in close proximity to the robot arm. This can
sufficiently distraction-robust for AURORA to maintain a stable diversity
environment does
be found in the setup of the arrangement scenario. In this task,
7

---

Under review as a conference paper at ICLR 2021
be seen in the depiction of the initial simulation state in Fig. 2 staged controller. Consequently, even when DIs distort the perception of the robot's behaviour, most discovered controllers will still produce a movement in at least one of the objects. This explains the AE variant's minimal proportion of no-move controllers across environment stochasticities.
Second, the robot executes a two-
Fig. Gp further shows that 0% environment stochasticity the of RAED in the air-hockey task re- sults in the same minimal proportion of no-move controllers as the use of the AE. However, RAED's score in the arrangement task slightly larger in Fig. GA. In this scenario, the AE and RAED vari- ants produce scores of 0.01 and 2.53 percentage points respectively. This disparity is explained by the larger number of parameters in the controllers used in the arrangement task. In a distraction- -ree environment, all no-move controllers generate the same observed behaviour. However, each con- troller has different parameters which serve as the inputs to RAED's s Encoder. These differences can therefore allow RAED to produce different BD for each of these no-move controllers. In this case, AURORA would believe these controllers to generate different behaviours. As result, a larger pro- portion of no-move controllers is accepted into the collection. In the air-hockey task, the controllers only have 4 parameter values. The variability in RAED's inputs is therefore much more limited than in the arrangement task, where controllers consist of parameter values which makes dispersion in the BDs more likely. However, one can note that RAED's marginally larger proportion of no-move controllers in the arrangement scenario does not affect the overall learning outcome of AURORA. The AE and RAED achieve the same variance scores at 0% environment stochasticity in Fi..3
One approach to reducing RAED's freedom to disperse the representations lies in the use of penalty term. In
we evaluate a VAE that takes the robot's 's actions as its inputs. The Encoder network now produces the mean and variance of normal distribution. As discussed in Section 3 we can interpret the KL divergence of this produced distribution and a standard Gaussian distribution as a penalty term on the size of the latent space. However, the results in Figs. and 6 show that the application of this penalty in the object arrangement scenario results in rapid deterio- ration in both metries as the environment stochasticity increases. This explained by the KI. term's pressure on the network to increase Encoder variances up to a magnitude of 1. Large variance values lead to greater dispersion in the learned representations when these are sampled from the Encoder. The Encoder variance term can therefore be interpreted as a further source of stochasticity in the generation of the learned representations. We therefore omit the BAED-KLPENALTT variant from evaluation in the air- air-hockey task. Instead, we create the RAED-LINPENAALTY configuration by removing the Encoder variances. This simplifies the KL term to the penalty term Piinear that we introduced in Section R In the air-hockey task, the introduction of this linear penalty term leads to a lower proportion of no-move controllers in Fig. 66 and a larger variance score in Fig. B In the arrangement scenario, on the other hand, the use of RAED and RAED-LINPENALTY produces ap- proximately the same scores in both performance metrics in Figs.Br 3 and (4. This suggests that the efficacy of this penalty term depends on the specifics of the task and the observation type. We can introduce a weight to control the effective cost of the penalty in order to adapt it to the task at hand. However, any result shown here on the optimality of a given magnitude in this weight would be specific to our experiments. We therefore omit any further evaluations of different weight values, as we want to avoid any such -pec-speyyyyccty in this work in order to preserve RAED's more generally applicable concept of using the robot's actions to learn representation.
5 CONCLUSION
In this paper, we introduced the concept of Distraction-Rodl Robust Representation Learning, We demon- strated the necessity of this approach in two experimental scenarios. The use of an Autoencoder led to large distortion in the robot's 's perceived behaviour when distracting influences were present in the environment. The susceptibility to environment distractions led to poor learning outcomes when using the Autoencoder's learned representations. We introduced RAED, an Encoder-Decoder archi- tecture which learns representations that allowed the generation of controller collections to become robust to the distracting influences considered in both scenarios. Finally, we also show that the num- ber of controllers that do not interact with the objects in the environment can be reduced with the introduction of linear penalty term in the loss function.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maya Cakmak, Mchmet R Dogar, Emre Ugur, and Erol Sahin. Affordances as a Framework for Robot Control. In Proceedings of the Seventh International Conference on Epigenetic Robotics, 2007.
Antoine Cully. Autonomous skill discovery with quality-diversity and unsupervised descriptors. GECCO 2019 - Proceedings of the 2019 Genetic and Evolutionary Computation Conference, 9: 81 89, 2019. doi: 10.1 10.1145/3221777.33218888
Antoine Cully and Yiannis Demiris. Hierarchical behavioral repertoires with unsupervised descrip- tors. GECCO 2018 Proceedings the 2018 Genetic and Evolutionary Computation Conference, pp. 69-76, 201 8a. doi: 10.1145/200455530055777
Antoine Cully and Yiannis Demiris. Quality and Diversity Optimization: A Unifying Modular Framework. IEEE Transactions on Evolutionary Computation, 22(2):445-259, may 20 8b. ISSN
1089778X. doi: 10.1109/TENC.2217.770011 09251
URL http://arxxv..org/abs/
Kalyanmoy Deb and Ram Bhushan Agrawal. Simulated Binary Crossover for Continuous Search Space. Complex Systems, 92)):133, 1994. ISSN 08912513. doi: 10.1.1.26.
8485Cached. URL htwpullkkeeartrcle-1111888111660
ast.psu..
8485
Atabak Dehban, Lorenzo Jamone, Adam R. Kampff, and Jose Santos-Victor. Denoising auto- encoders for learning of objects and tools affordances in continuous space. In Proceedings IEEE International Conference on Robotics and Automation, volume 2016-June, pp. 4866-4871. Institute of Electrical and Electronics Engineers Inc., jun 2016. ISBN 9781467380263. doi:
10.1109/ICR..2166 7487691.
Andrej Dobnikar, Nigel C. Steele, David W. Pearson, Rudolf F. Albrecht, Kalyanmoy Deb, and Samir Agrawal. A Niched-Penalty Approach for Constraint Handling in Genetic Algorithms. In Artificial Neural Nets and Genetic Algorithuss, pp. 235-243. Springer Vienna, 1999. doi: 00.1007/978-3-701--655
40. URL hhttp::/ springer. com/ chapter/1c. 1007 709-8----7791-38------ 9384-999----999-11 -0.
Benjamin Eysenbach, Julian Ibarz, Abhishek Gupta, and Sergey Levine. Diversity all you need: Learning skills without reward function. In 7th International Conference on Learning Repre- sentations, ICLR 2019. International Conference Learning Representations, ICLR, feb 2019.
URL https: /ttp::://ssteee tes http://sssss..s.llle.....oo www.glle....eeeooo wwom/view/ddaym
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In Proceedings IEEE International Conference on Robotics and Automation, volume 2016-June, pp. 512-519. Institute of Electrical and Electronics Engineers Inc., 2016. ISBN 9781467380263, doi: 10. 10.110//A....1......0 7.71167.7777....
Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Fierre-yves Oudeyer. Intrinsically Moti- vated Goal Exploration Processes with Automatic Curriculum Learning. 2017. URL http:
//axxiv.rrr/abb//70000 0211/
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 31365786):504-007, jul 2006. ISSN 00368075. doi: 10. 126//cience. 127647.
Michael Jordan and David Rumelhart. Forward Models: Supervised Learn-
ing with Distal Teacher. Cognitive Science, 16(3):307-355 jul 1992. ISSN 03640213, doi: 10. 1207// s15516709com 603_1
PRLLhttp:/ddi.witr
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. 3rd
International Conference Learning Representations, ICLR 2015 Conference Track Proceed- ings. International Conference on Learning Representations, ICLR, dec 2015. URL hetps:
T/arxiv.oro //arxiw..rrg/ab///////
/ttp///aabbb111...6998999 6980v9

---

Under review as a conference paper at ICLR 2021
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Con- ference on Learning Representations, ICLR 2014 Conference Track Proceedings. International Conference on Learning Representations, ICLR, dec 2014.
Sascha Lange. Martin Riedmiller, and Arne Voigtländer. Autonomous reinforcement learning on raw visual input data in real world application. In Proceedings of the International Joint Conference on Neural Nenworks, 2012. ISBN 9781467314909, doi: 10.1 1LL00//CCN..2012..222222222
Adrien Laversanne-Finne Finot, Alexandre Péré, and Pierre Yves Oudeyer. Curiosity driven exploration of learned disentangled goal spaces, arXiv preprint 1rXivv1807.01522 2018.
Timothée Lesort, Natalia Díaz-Rodríguez, Jean Franois Goudou, and David Filliat. State repre- sentation learning for control: An overview. Neural Networks, 108:379-392, dec 2018. ISSN 18792782. doi: mww.e///...mmet2001...... 200.000006
Tanis Mar, Vadim Tikhanoff, Giorgio Metta, and Lorenzo Natale. Self-supervised learning of grasp dependent tool affordances on the iCub Humanoid robot. In Proceedings IEEE International Conference on Robotics and Automation, volume 2015-June, pp. 3200-322 -2006 Institute of Electri- cal and Electronics Engineers Inc., jun 2015. doi: 10.1 1../10//CRAAA 2011//CAA...1......... 201...211.....
Jean-Bad Mann-Batistte Mouret and Stéphane Doncieux. Sferes v2 Evolvin' in the Multi-Core World. In Proc. of Congress on Evolutionary Computation (CEC), 2010. URL
gel. gel.
ulaval.cam
Jelle Munk, Jens Kober, and Robert Babuska. Learning state representation for deep actor-critic control. In 2016 IEEE 55th Conference on Decision and Control, CDC 2016, pp. 4667-4673. Institute of Electrical and Electronics Engineers Inc., dec 2016. ISBN 9781509018376. doi: 10.1 10.119//DDC220016779999
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In 34th International Conference on Machine Learning, ICML 2017, volume 6, pp. 4261-4277 International Machine Learning Society (IMLS), may 2017. ISBN 9781510855144 URL
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-Aware Unsupervised Discovery of Skills. 8th International Conference Learning Representations, ICLR 2020, 2019. URL htthtt/p//axixii orge//iiv..rg/eee htspgg/ab/11999/ 190110066777777
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Ku- mar, and Sergey Levine. The Ingredients of Real-World Robotic Reinforcement Learning. In 8th International Conference Learning Representattinss, ICLR 2020 Conference Track Pro- ceedings, 2020. URL http:::/sssssttteeess gowgle..ooooeccooooooooo http://eeoooogeeeeoo///////eeala
httttpp:::
7/arxiv. wwtp://wwgabbbb/.....o/ 12570
Erol Şahin, Maya Çakmak, Mehmet R. Dogar, Emre Uğur, and Gktrk Üçoluk. To afford
to afford: A new formalization affordances toward affordance-based robot control. Adaptive Behavior, 1544)4447472, dec 2007. ISSN 10597123 123. doi: 10. 10.11777100971120000666666 URL
ctt http://ow...aaaaaaoooooooooo....
10

---

Under review as a conference paper at ICLR 2021
A NETWORK PREDICTIONS
We can inspect the networks' predictions to obtain qualitative assessment of the learned representa- tions' distraction-robustness In the following visualisations, we show the networks' predictions and the corresponding environment observations in the air-hockey task 100% environment stochastic- ity. In this task, the presence of DIs materialises an additional object in the environment. This allows clear distinction of the observation elements that are present due to the DIs. In the object arrangement task, the DIs are more difficult to distinguish in the observations. We therefore do not show any visualisations from that experiment scenario here.
Construction
Observatio on
corstruction
Ooservation
Figure 7: Air-hockey scenario at 100% environ- ment stochasticity. Left: Prediction made by RAED. Right: Corresponding environment ob- servation showing a particularly misleading en- vironment observation. The distinction between the "actual" puck and the random puck is made for analysis purposes and available the net- work. Pixel activations range from to I and are thresholded a value of 0.5 for the illustration.
Figure 8: Air-hockey scenario at 100% environ- ment stochasticity. Left: Prediction made by RAED. Right: Corresponding environment ob- servation showing a collision between the two pucks. Pixel activations are thresholded as in Fig. 7
Fig. 7 shows the environment observation and associated prediction made by RAED. The distinction between the actual non-random puck and the random puck in the observation is made only for analysis purposes and is not available to the network. All pixels take values in the range of [O, 1]. The prediction visualisation is thresholded
value of 0.5. The environment observation shows that the two pucks' trajectories connect to form a longer trajectory. This type of physically plausible observation is one of the most difficult types of observations for RAED predict. However, can see that RAED's prediction accurately traces out the non-random puck's 's trajectory, while ignoring the random puck entirely. Furthermore, by encoding the expected perceived consequences of the robot's s actions, RAED is able to impact of collisions between the two pucks. Fig. & shows a collision between the two pucks at the table coordinates (1, 3). Nevertheless, RAED's prediction accurately traces out the expected trajectory instead.
Krecn
1
Figure 9: Air-hockey scenario at 100% environment stochasticity. Left: Prediction made by RAED. Right: Corresponding environment observation showing the non-random puck colliding with the robot arm. Pixel activations are thresholded as in Fig.
Another challenging situation for RAED is posed by the puck colliding with the robot Small changes in the controller can lead to large differences the shape of the arm and therefore the trajectory of the puck after impact. For these cases, observe in the worst case that RAED's un-

---

Under review as a conference paper at ICLR 2021
certainty results in truncated trajectory prediction. We show example in Fig. Note, however, that RAED successfully excludes the trajectory the non-random puck from the prediction.
Construction
Ooservation
Construction
Observat on
Acuul
Figure 10: Air-hockey scenario at 100% envi-
Figure 11: Air-hockey scenario at 100% envi-
ronment stochasticity. Left: Prediction made by
ronment stochasticity. Left: Prediction made by AE. Right: Corresponding environment observa- tion showing two unconnected puck trajectories. Pixel activations thresholded in Fig. 7
AE. Right: Corresponding environment observa- tion showing two unconnected puck trajectories. Pixel activations are thresholded in Fig.
The AE method's predictions are strongly affected by the presence of DIs. Every prediction we inspected, features elements of the non-random puck's trajectory. In the best case, the two trajecto- ries are not connected and the non-random puck trajectory is not reconstructed with full certainty. Fig. shows example of this. This is due to the limited variability in the achievable trajectories of the non-random puck. Certain areas of the table are frequented largely or even exclusively by the random puck only. The low frequency of observations that feature activated pixels in these areas, allows the AE to ignore the reconstruction of some of these observation elements. However, in the worst case, the non-random puck transverses environment observations. Fig. [[] shows that in these situations, the AE accurately reconstructs the full trajectory of the non-random puck. This demonstrates the strong susceptibility of the AE's learned representations to the DIs in the environment.
of the table that frequently activated in the
B IMPLEMENTATION DETAILS
B. AURORA
We provide pseudo-code of AURORA Algorithm|] [ Tab D presents the hyperparameters used in the configuration of AURORA. Our AURORA implementation is built using the Sferes2 framework Mouret & Doncieux 2010).
12

---

Under review as a conference paper at ICLR 2021
Algorithm Pseudocode the AURORA algorithm (Cully 2019).
1: procedure
2: collection +- random controllers Initialise collection with set of random controllers
3: L iniiialiis Accolection)
Initialise distance threshold I.
4:
for iteration in num_iteratioon do
5:
parents + celecicollecctin))
Select set of parent controllers
6:
new sew.tonrrllsers + connutttecroos.ove(pprreee generate (DrraateBBD(eeeaee (new (en(ecc..oot))lllr))))
7:
Use AE or RAED to generate BDs
8:
ADD (collection, new cew.controller)
9:
if iteration % braininggfrequeeee
then
10:
www..nelwuokkcolllclteei all. comtrolllers &- cadtracl_allccolleett
12:
generate. o .oll.ccotrlleers)
13:
ADD (collection, contrrorlll))))
14:
+ update disstance
collection, collection.targel.ssii.
15:
16:
procedure ADD(callection.comnl controllers) for controller in controllers do distance & get_BD wtw.eeee..oerrretcoooooooo if distance L then [.l.lo.collcciionconnoollle
D Attempt to add controllers to collection
17:
18:
(controolor,. collection)
Add controller if sufficiently novel
20:
Parameter
Value
Behavioural Descriptor Dimensionality
Mutation Rate
0.1
Cross-Over Rate
0.
Mutation Operator
Polynomial (Dobnikar al.
1999)
Cross-Over
Cross- Over Operator
SBX (Deb & Agrawal 1994)
Tm (Mutation Parameter)
15
Te (Cross-vver Parameter)
15
Collection Target Size
8000
AURORA controller Range of Possible Values
[0, 1]
Rehavioural Descriptor Size
Number of Newly Generated controllers each Iteration
256
First AE/ RAED Training iteration
10th Iteration
Frequency of AE RAED Training
Every 10 Iterations
Number of Iterations (Object Arrangement Task)
4000
Number of Iterations (Ai--Hockey Task)
15000
Table 1: Hyperparameters for the AURORA implementation.
13

---

Under review as a conference paper at ICLR 2021
B.2 AE AND RAED HYPERPARAMETERS
In the object arrangement scenario, the AE is made up of convolutional layers including the output layer in both the Encoder and Decoder network. RAED's Encoder consists of 3 fully connected lay- ers. The Decoder in RAED consists of 6 transposed convolutional layers. The network architecture
details are given in Tab.
Layer
Number of Filters Neurons
Kernel Size
Stride
AE Encoder #1
2
9
9
AE Encoder #2
4
8
x
2
AE Encoder #3
7
x
AE Encoder Output Layer
2
7
AE Decoder #1
6
7 x
AE Decoder #2
7
AÉ Decoder #3
2
8x8
2
AE Decoder Output I Layer
1
9x
RAED Encoder
10
RAED Encoder #2
RAED Encoder Output Layer
2
RAED Decoder #1
10
3 x
RAED Decoder #2
10
3 3
RAED Decoder #3
20
4x
2
RAED Decoder #4
5 5
RAED Decoder #5
40
5x
2
RAED Decoder Output Layer
6
1
Table 2: AE and RAED architecture details.
In the air-hockey scenario, the AE Encoder consists of 6 convolutional layers The AE Decoder features 6 transposed convolutional layers. RAED's Encoder is made up of 7 fully connected layers and its Decoder contains 9 transposed convolutional layers. The architecture details used in the air-hockey task are given in Tab.
Layer
Number of Filters Neurons
Kernel Size
Stride
AE Encoder #1
10
6x
AE Encoder #2
10
5
2
AE Encoder #3
5
AE Encoder #4
20
4x
2
AE Encoder #5
30
3
1
AE Encoder Output Layer
3
RAED Encoder #1
20
RAED Encoder #2
30
RAED Encoder
RAED Encoder #4
40
RAED Encoder #5
30
RAED Encoder
RAED Encoder Output Layer
2
RAED Decoder #1
10
2x
RAED Decoder #2
3
RAED Decoder #3
20
x
RAED Decoder #4
20
x
RAED Decoder #5
x 3
RAED Decoder #6
40
3 x
RAED Decoder #7
60
5 x
RAED Decoder #8
2
RAED Decoder Output Layer
6 6x6
Table 3: AE-SMALL and RAED-LARGE architecture details.
14

---

Under review as a conference paper at ICLR 2021
Both networks trained with the same training parameters given in Tab. implemented using the PyTorch library m
The networks
Parameter
Value / Definition
Batch Size
256
Learning Rate
0.0001
Optimisation Algo-
Adam (Kingma & Ba 2015)
rithm
Adam Betas
(0.9, 0.999)
Validation Set Size (% of total dataset)
80%
Maximum Number
5000
of Training Epochs
Early Stopping Crite-
Validation error larger than running mean of validation errors over 5 epochs AND most recent training loss smaller than at start of training
rion
Table 4: Network training details.
C SIMULATION DETAILS
Our simulated environments utilise the Box21 physics simulation engine to model robot arm mounted in the center of table. The arm is controlled with servos setting the motor speed at each joint to reduce the angle error. We specify the Box2D specific simulation parameters in Tab. [S] The Box2D source files were modified to change #define macro, to allow for simulated pucks to bounce instead of gliding along any walls after impact. The change was made in file hox2d//ncluue/hox2dd// settings.h to update the value of #define b2 velacityThreshole from 1.0f to 0.0f.
Configuration Element
Dimensions (half sizes per Box2D convention) / Value
Gravity
(0, 0)
Robot Base
0.0375 x 0.0375
Arm Segment
0.1875 0.015
Arm Segment #1 Density
Arm Segment #2 Density
0.666666
Arm Segment #3 Density
0.444444
Arm Sogment #4 Density
0.296296
Arm Friction (All components)
0.8
Servo Maximum Motor Torque
Table Dimensions
x 5
Table Wall Friction
0.8
Simulation Duration
10 seconds
Environment Object Radius
0.25
Environment Object Friction
0.8
Environment Object Restitution
0.8
Scenario Puck Density
0.2
Scenario Maximum Random Force
Scenario 1 Ron-Random Puck Start Position
(3.55, 3)
Scenario 2 Object Density
0.8
Scenario 2 Maximum Force
70
Scenario 2 Linear Damping
1.8
Scenario Object Start Position
(3.55, 3)
Scenario 2 Object 2 Start Position
(2.55, 1.7)
Table 5: Box2D configuration parameters.
15


Reviews:
[{"review_id": "yMQuUFVZ8H0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper seeks to learn a representation space for trajectories that discards state elements that are not affected by the agent. To do so, it proposes an objective to predict the last observation in a trajectory from the sequence of actions taken in the trajectory.\n\nPros:\nTackles the problem of generating diverse task-relevant behaviors\nCons:\nThe motivation and the presentation of the method are very confusing\nI am not convinced that the objective is a reasonable one (see comments below)\nExperiments are very toy, and the analysis is hard to parse\n\nDetailed Comments\nIntroduction: “When the dimensionality of the representation is smaller than that of the input data, some information is lost when creating the representations.” Dimensionality reduction does not necessarily imply information reduction. Also, the goal of representation learning need not be dimensionality reduction (sparse coding, for example).\n\nI do not think that terms such as “representation learning” and “reinforcement learning” should be capitalized. They are not proper nouns.\n\nIt’s strange that “a” refers to a sequence of actions, but “x” refers to a single observation.\n\nIt seems like in Equation (2) you would want to predict a distribution over final states, to account for stochastic dynamics. Also, it seems like there is an implicit assumption that all objects need to start in the same place. For example, say you want a robot to pick up a cup. Depending on the starting position of the cup, the same sequence of actions could lead to very different final states.\n\nThe discussion in Section 3 about choosing the form of the penalty P is quite confusing and not rigorous.\n\nIt’s not clear until the experiments section of the paper that the goal is to produce diverse trajectories. This makes the beginning of the paper very hard to understand.\n\nHow is the representation learned by the auto-encoder of each observation transformed into a behavioural descriptor for the whole trajectory? Is the behavioural descriptor simply taken to be the embedding of the final observation? If this is the case, then how do the DIs affect the auto-encoder baseline at all?\n\nThe air-hockey plot seems to be missing a line?\n\nRecommendation:\nReject\nThe method as presented seems flawed, and the experiments are toy and hard to make sense of. The paper overall is very difficult to understand. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WWVKMK-P86", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper approaches the problem of representation learning for robotic learning, with a focus on being robust to potentially misleading distractor objects. Concretely, they propose a method which learns a representation of sequences of actions by reconstructing a future state, and uses this representation of actions to learn skills. They show that this leads to some improvement in identifying more diverse \"skills\" in settings where there are external distractors.\n\nPros:\n+ The problem of learning representations for robotic learning which are robust to distractors is very important and relevant. \n+ The proposed approach is simple to implement.\n\nCons:\nI have several concerns about the paper, concretely regarding (1) the clarity of presentation, (2) the correctness/generality of the proposed method, and (3) the conclusions drawn from the experimental results.\n\n(1): There are serious issues with the clarity of the paper. \n\nFirst, much of the abstract/introduction motivates the challenge of learning representations of visual inputs that are robust to distractors. The related work also covers some prior works on this topic in general the paper seems to be proposing a technique to address this challenge. But upon reaching the method, it appears that the paper is not concerned with representation of states at all, but rather representations of sequences of actions with no state information. This a completely disconnected from the motivation/problems presented in the introduction/related work. It is not clear what the objective of the paper is - is it to learn efficient state representations for reinforcement learning? is it to learn a generative model over actions? What is the goal of such a representation, better reinforcement learning? These questions should be clearly answered early in the paper.\n\nSecond, this representation of action sequences is not actually used for any specific RL or task learning. Instead it is used with a method for unsupervised skill discovery AURORA, which tries to explore + build a set of controllers. If the motivating problem is representations robust to distractors, why use a unsupervised skill discovery framework like this? It seems like the problem of robust representations can be studied in a much simpler way (evaluating some task performance in the face of distractors). On the other hand, if the goal is a better exploration + skill discovery framework using a compact action representation, that should be laid out from the beginning of the paper.\n\n(2): I have some doubts about the correctness of the method (or the implicit assumptions made). \n\nFirst, The idea of the paper is to encode a sequence of actions into a latent distribution from which the future observation can be recovered. Is this not simply learning a forward dynamics model, but under the very restrictive assumption that the future state does not depend on the current state? Is this work considering an MDP? The necessary preliminaries are not stated, so its not clear what problem setting the paper considering.\n\nSecond, the way the KL penalty is used on the latent of action sequences does not seem correct. The encoder is outputting a latent distribution over action sequences. The reason to have this be a distribution is to capture stochasticity in the learned representation. But the authors then argue they don't want the latent distribution to have high variance, so they only use a penalty on the mean. In that case, why have the encoder output a distribution?\n\n(3): Experimental conclusions.\n\nThe only experimental comparison is the an auto encoder on state. Since the proposed method is not encoding state, it seems obvious that the representation would be robust to state perturbations, while the auto-encoder on the state would not be. In fact its not clear why the auto-encoder on state is an effective baseline, if the goal is to learn sets of controllers composed of action sequences. Wouldn't simply learning an auto-encoder on action sequences be a more effective baseline?\n\nAlso more of a minor point, but the points to not considering realistic/stochastic settings with distractors as a limitation of prior work, but the domains considered in this paper are far from realistic. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lhygpj3Hum8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nDistractions are alterations of states or observations outside the control of the agent.  Conventional learning methods tend to be quite sensitive to them.  This work proposes a method for learning representations robust to such distractions using an encoder-decoder architecture whose encoder uses actions as input, instead of an observation as conventional autoencoders do.  In addition, the paper evaluates two variants of this method that employ an additional regularization term, one being the KL divergence as used by variational autoencoders, the other the $L_2$ norm of the latent activation vector (to avoid penalizing its variance as the KL divergence does).\n\nThe method is evaluted as the generator of the latent space (\"Behavioral Descriptors\") of an AURORA (Cully 2019) setup where randomly-evolved controllers are accepted or rejected based on their diversity in latent space.  Two example tasks using a planar, 4-DoF robot arm and 40$\\times$40 image observations are used. The first task involves pushing two high-friction objects; distractions are externally-imposed displacements of one object.  The other task involves hitting one low-friction object (air-hockey style); distractions consist in the insertion of a second, random puck trajectory into the observation.  By most metrics, the method is shown to be effective at increasing robustness of the latent representation to distractors.\n\n#### Strengths and Weaknesses\n\nStrengths include:\n\n* The stated problem is important and arises in many contexts.\n* The key part of the method, learning representations by encoding actions as opposed to observations, makes a lot of sense in this context.\n* The proposed solution can be plugged into existing representation-learning methods, as is done in the evaluation.\n* For the most part, the paper is well organized, well written, and clear.\n* The appendix provides implementation details.\n\nWeaknesses include:\n\n* As described, the method can only be used to work with controllers that operate on a state space distinct from the learned latent space.  Thus, the latent space can only used for purposes other than control, such as selecting controllers based on diversity criteria.  In practice, one usually wants to use distraction-robust latent representations as input to (distraction-robust) controllers.  This work does not appear to be applicable to this use case.  Thus, the practical scope and usefulness of this method appear to be rather limited, and the ultimate value of the learned representation is unclear.  In this light, the title and abstract can easily be misread as promising more than the article delivers (\"We propose an Encoder-Decoder architecture that produces representations that allow the learning outcomes of robot control tasks to remain unaffected by these distractions.\").\n* In Figure 6b, the RAED methods perform rather poorly in absolute terms.  No explanation is given (besides an explanation of why the AE method performs relatively well here).\n* The evaluation considers two quite similar tasks, and the results are not fully conclusive.  Thus, it is hard to extract a clear take-home message from this paper.\n* As formulated, the input vector to the encoder represents an action sequence encoding an entire episode.  In the experiments shown this did not pose a problem as an episode consisted of a single action.  However, it is unclear how the method would perform on multi-action tasks.  Encoding entire episodes would not scale and is not possible on non-episodic tasks.  It is unclear how the method would be used and how it would perform if applied on single actions in multi-action tasks.\n* The technical contribution is limited, quite close to the smallest publishable unit (if such is quantifiable), especially considering the inconclusiveness of the experimental results.\n\n#### Justification of Recommendation\n\nThe technical contribution is minor, its applicability is quite limited, and the results are not fully conclusive.  This weakens the take-home message of the paper.\n\n#### Questions for the Rebuttal\n\n* Please comment on the weaknesses listed above.\n* In Sec. 4.1, observations are taken at the *end* of a simulation, and DIs are injected at the *start* of a simulation.  Does this mean that the disturbed state is never directly observed but manifests itself only indirectly at the next simulation? That is, the state persists across simulations and is not (randomly) initialized for each simulation?\n* Figures 3 and 6 show interquartile ranges. Since these are computed over only 5 repetitions, I take this to mean that the ranges shown exclude the minimum and the maximum value everywhere.\n* In Fig. 6b, *failing to move both objects* is apparently intended to mean *failing to move one of the two objects*.  The latter formulation is more clear, as the former logically includes failing to move any object.\n* Why was RAED-KLPENALTY excluded from evaluation on the air-hockey task? I don't think the reason given justifies making the results appear incomplete.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LYHIyOCzylU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the concept of Distraction-Robust representation learning in their words. Their algorithm embeds autoencoder (which weirdly takes action as input. to restore images) into the AURORA framework(a kind of genetic algorithm). They conduct two toy experiments to show the effectiveness of their algorithm.\n \n\nStrenght:\n- The paper is well-written and easy-to-follow.\n- The paper is well-motivated. Learning a distraction-robust representation is important for robot vision tasks, e.g. object tracking.\n\nWeakness:\n- This work lacks novelty. The core algorithm is just a regular autoencoder that has been well studied, see [1] for a thorough clarification. And the framework of the genetic algorithm mainly draws on AURORA. \n- The toy experiments are not convincing. The setting is relatively clean/simple, comparing to most real-world scenarios, in the aspect of system dynamics, appearance, and etc. For the robot learning field,  the tasks should be defined closely to real-world robot setting such as object manipulation[2], object tracking[3], etc, either in a simulation environment or the real world.  In my opinion, it is necessary to conduct an experiment to demonstrate the generalization of the proposed method. For example, the simplest idea is training/testing the model in environments with random nature noise (placing different nature images/videos as the background).\n\nRef:\n\n[1] Kingma D P, Welling M. An introduction to variational autoencoders[J]. arXiv preprint arXiv:1906.02691, 2019.\n\n[2] Wilson, Matthew, and Tucker Hermans. \"Learning to manipulate object collections using grounded state representations.\" Conference on Robot Learning. PMLR, 2020.\n\n[3] Luo, Wenhan, et al. \"End-to-end active object tracking and its real-world deployment via reinforcement learning.\" IEEE transactions on pattern analysis and machine intelligence 42.6 (2019): 1317-1332.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the general idea of the paper is certainly interesting and highly relevant, there is consensus that the paper cannot be published in the current form.
There were serious concerns about
- the correctness and generality of the method
- clarity of presentation
- experimental evaluation

The authors graciously accepted the feedback, we wish them all the best in thoroughly revising and resubmitting the paper.
Decision: Reject