In this work, we present the key point-based neural rendering approach for dynamic shapes called Neural Puppeteer. Post-Systemation is one of the fundamental problems in computer vision and the crucial task in many vision applications, ranging from sports to human computer interaction to the study of collective behavior. Existing models learn how joints look like. Among other things, they learn the texture of the joints. To drain such models, a huge amount of data with different textures is available for humans. Data sets for animals have been published too. Recently, these data sets cover an increasingly wide range of species ranging from mice to monkeys to birds. Our approach called Neural Puppeteer on the other hand is based on an efficient and flexible rendering pipeline that renders image properties like occupancy, depth, and texture given 3D key points. In case you are wondering how Neural rendering and post-Systemation connect the answer is inverse rendering. In case of Neppu, we can even estimate 3D key points from multi-fuse silhouettes only, making our post-Systemation method independent of texture. Let us now take a closer look at how Neppu works step by step. For our inverse rendering solution, it is crucial to learn the relations of cores, defined by 3D key point positions, and resulting to the image properties such as RGB, silhouettes, and depth. Once such a prior is learned, we can later use it to infer 3D key points from given to the observations. In the next slides, we explain our differentiable forward rendering step by step. The first part of our model encodes 3D key points in the latent variable Z using methods from deep learning on point clouds. Namely, we rely on the vector self-attention operator proposed in point transformer. From a high level perspective, the vector self-attention mechanism facilitates information exchange between nearby key points. Therefore, it solves a similar purpose as convolutions and CNNs. Our decoder first decompresses the single latent vector into key points and local features in two separate branches. The purpose of the local features is to describe the surroundings locally. We then refine the features using the same attention operator as before. We incorporate camera parameters of the rendering procedure by projecting 3D key points into image space alongside the local features. In Tudy, we facilitate the final round of attention for further refinement. Importantly, we perform rendering directly in Tudy since volumetric renderings took costly for our application scenarios. For example, a neural radiance field, it is common to perform 64 to 128 network queries per array, while we only need a single query. As we will later see, our model still learns good view consistency, despite the projection of key points as a much weaker view consistency constraint than a volumetric rendering formulation. Finally, image information for our query pixel position queue is obtained by first gathering information from surrounding local features. For this, we rely on the cross-attention mechanism introduced in ANATS to aggregate nearby information from which the final predictions are made. Our model has three multi-layer perceptron heads, one for each different modality. In our case, we have one rendering head for RGB, one for depth, and one for occupancy. All network parameters are trained jointly using a loss consisting of three parts. First, we have a key point reconstruction part. Second, we have a rendering loss between the color, depth, and occupancy masks. And additionally, we regularized latent variables in the norm, which roughly encourages a zero-mean Gaussian latent distribution. Since available data sets for animals do not provide all the necessary information, we decide to explore our method on synthetic data. For this purpose, we generated high-quality data for four different specimens, cows, giraffes, pigeons, and humans. Our data set has 24 post-camera with high-quality ground truth, RGB, depth, and masks, and time synchronized 3D key point locations. In our inverse rendering experiments, we aim to infer 3D key point positions from multiple silhouettes. Since our end goal would be an easy domain transfer from synthetic to real, we only rely on 2D occupancy masks as observations, which makes our 3D key point detector invariant to texture and illumination changes. To solve this highly complex inverse rendering problem, we heavily rely on the prior distribution that may be learned over the relations of 2D image properties and 3D key points. Using this prior, we aim to find a latent vector z-hat that minimizes the binary cross entropy over all cameras and pixels between the observed and rendered masks. Additionally, we include a regularization term on z in order to arrive at the more likely and therefore more well-behaved solution. Using our decoder, we can ultimately infer the 3D key points from z-hat. Since our optimization sometimes gets stuck in local minima, if the initialization is too far away from the true latent variable, we perform multiple runs with different initializations and take the one with the highest intersection over union between the rendered and ground truth masks. This animation shows the process of post-optimization from the initialization to the final post given the silhouette. Please note that these are two of our zero-shot synthetic to real-world experiments where we train NAPO on our synthetic data and then inferred on real-world images. Here we see some more real-world examples for giraffes and the cow. Our inverse rendering for post-estimation works very well and we achieve a better average mean-per-joint position error and median over all objects than learnable triangulation of human posts. Our qualitative results reflect the overall good quality of NAPO for post-estimation. We show projected key points where color-coded dots indicate a location and a 3D error to the ground truth. Red dots indicate a high, green dots allow 3D error. Our forward-differentiable rendering pipeline works very well too and we achieve a better average peak signal to noise ratio and mean absolute error over all objects than light field networks. Our qualitative results reflect the overall good quality of NAPO for a forward-differentiable neural rendering. Comparing the results shows the importance of the local conditioning. The results obtained with NAPO are much more detailed with more precise boundaries. We also compare NAPO to ANI-NIRF. Our results look promising but cannot meet the quality of ANI-NIRF. Please note that our rendering formulation is fundamentally different. While ANI-NIRF leverages the simple parameters to restrict the problems to computing plant weights, NAPO has to solve a more complex problem. In addition, NIRF based approaches typically rely on time and memory intensive volumetric rendering. In contrast, we propose a single evaluation rendering formulation that is closer to the formulation of light field networks. This fundamental difference in our rendering pipeline results in a significant speed increase of 50 to 200 times at an image resolution of 5 on the 12 times 5 on the 12 pixels compared to NIRF based approaches. Furthermore, we applied our rendering pipeline to key points from motion capture data from the AMAS dataset. Wow, NAPO can even dance Macarena. Also, the 2D depth estimations from NAPO can be used to perform 3D reconstruction of the capture shape. Although NAPO renders into D, which means that we do not have a guaranteed consistency between the generated views, we observe a very good view consistency. And last but not least, the structure of NAPO's latent space and codes meaningful information about how individual poses are related to each other. For code, our synthetic dataset and further results, please visit our project page by scanning the QR code.