TASK DEFINITION: In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.
PROBLEM: Many real-world sequential decision-making problems can be formulated as optimal control with high-dimensional observations and unknown dynamics. A promising approach is to embed the high-dimensional observations into a lower-dimensional latent representation space, estimate the latent dynamics model, then utilize this model for control in the latent space. An important open question is how to learn a representation that is amenable to existing control algorithms? In this paper, we focus on learning representations for locally-linear control algorithms, such as iterative LQR (iLQR). By formulating and analyzing the representation learning problem from an optimal control perspective, we establish three underlying principles that the learned representation should comprise: 1) accurate prediction in the observation space, 2) consistency between latent and observation space dynamics, and 3) low curvature in the latent space transitions. These principles naturally correspond to a loss function that consists of three terms: prediction, consistency, and curvature (PCC). Crucially, to make PCC tractable, we derive an amortized variational bound for the PCC loss function. Extensive experiments on benchmark domains demonstrate that the new variational-PCC learning algorithm benefits from significantly more stable and reproducible training, and leads to superior control performance.   Further ablation studies give support to the importance of all three PCC components for learning a good latent space for control.

SOLUTION: Learning embedding for control with high-dimensional observations

PROBLEM: This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards. For example, we report a state-of-the-art score of >11,000 points on Montezuma's Revenge without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

SOLUTION: We investigate contingency-awareness and controllable aspects in exploration and achieve state-of-the-art performance on Montezuma's Revenge without expert demonstrations.

PROBLEM: We introduce the notion of property signatures, a representation for programs and
 program specifications meant for consumption by machine learning algorithms.
 Given a function with input type τ_in and output type τ_out, a property is a function
 of type: (τ_in, τ_out) → Bool that (informally) describes some simple property
 of the function under consideration. For instance, if τ_in and τ_out are both lists
 of the same type, one property might ask ‘is the input list the same length as the
 output list?’. If we have a list of such properties, we can evaluate them all for our
 function to get a list of outputs that we will call the property signature. Crucially,
 we can ‘guess’ the property signature for a function given only a set of input/output
 pairs meant to specify that function. We discuss several potential applications of
 property signatures and show experimentally that they can be used to improve
 over a baseline synthesizer so that it emits twice as many programs in less than
 one-tenth of the time.

SOLUTION:
We represent a computer program using a set of simpler programs and use this representation to improve program synthesis techniques.