SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use an off-the-shelf pre-trained Proximal Policy Optimization (PPO) controller [Schulman et al., 2017] from stable-baselines3 [Raffin et al., 2019] to compute a probability distribution over actions. Inference in this domain is complicated by the fact that computing backward dynamics in physical simulation is challenging and often ill-posed. While previous work has proposed analytic approaches [Twigg and James, 2008], we instead train a neural network to approximate the reverse physical dynamics. We place a unit Gaussian prior over the velocities, and use a von Mises distribution as a prior over the initial pole angle. We infer the velocities of the system by sampling candidate pairs of cart and pole velocities (stratified in an  $11 \times 11$  grid) and computing likelihoods using our algorithm.
The inferred posteriors are intuitive and track the relative stability of the position in each snapshot (Figure C.3). For example, in part (a), the pole has almost completely fallen over, and so our method infers that the pole has a large negative angular velocity, and is falling fast towards the ground. At the same time, it infers that the cart is moving fast to the left, because the controller is likely attempting to re-balance. In comparison, for part (f), the pole is nearly upright, so the model predicts that the pole is not rotating, and that the cart might be moving left or right to keep the pole balanced.
Figure 10: In each pair, the left image shows the cart-pole snapshot given to the algorithm, and the overlaid arrows summarize the model's predictions about how the system might evolve. The right heatmap shows our model's full joint distribution of inferred cart velocity (positive means moving to the right) and pole angular velocity (positive means clockwise), and the white stars mark posterior expectations. When the pole is near-horizontal, our algorithm infers that the pole is falling, and the cart is moving left to re-balance. When the pole is near-vertical, the algorithm infers that the pole is stationary, and the cart is making minor adjustments to keep the pole balanced.