---
author:
- 'Misha Wagner[^1]'
- 'Hector Basevi[ ^fnsymbol[1]{}^]{}'
- Rakshith Shetty
- Wenbin Li
- Mateusz Malinowski
- Mario Fritz
- Ales Leonardis
title: |
    Answering Visual [*What-If*]{} Questions:\
    From Actions to Predicted Scene Descriptions\
    Supplementary Material
---

18SubNumber

Data set and model output variation
===================================

We evaluate the vocabulary of the dataset and the vocabulary of the model output by forming “multi-level pie charts” to display the distribution of vocabulary used, showing only symbols taking up over 0.1% of the pie chart.  These are shown in Figure \[fig:vocabulary\_comparison\] of the supplementary material. 

The visualisation shows a variety in sentence structure for the data set.  However, the model learns the most common sentence structures in order to describe the effect of an action on an object, losing some of the variation in the human vocabulary. 

![ Comparison between the vocabulary of the ground truth effect descriptions and the hybrid model output.  Segment circumference represents frequency of the word occurring after the parent segment in the inner layer.  Instances of “the” as the first word in a sentence have been removed (a negligible amount of descriptions do not start with “the”), and visualisation depth is capped at eight layers/words.[]{data-label="fig:vocabulary_comparison"}](vocabulary_comparison){width="\textwidth"}

Replication details for networks
================================

The majority of important implementation details are shown in Figure 2, Figure 3, and Figure 4 of the paper.  They describe the flow of information inside and between each component of the hybrid model but abstract away key features, such as model hyperparameters and activation functions.  These are shown in Table \[table:hyperparameters\] of the supplementary material. 

For loss functions, all classification models (action type, acted object, push rotate parameters, and drop parameters) use categorical cross entropy, while other models (push parameters and description model) use mean squared error.  All models were terminated after 10 consecutive epochs of no improvement in validation error. 

[2]{}

-   Action type model

    -   Use pre-trained word embeddings

    -   Word embedding size of 200

    -   LSTM size of 240

    -   No dropout

    -   Softmax activation for probabilities over action types

-   Drop parameters model

    -   Use pre-trained word embeddings

    -   Word embedding size of 100

    -   LSTM size of 55

    -   No dropout

    -   Softmax activation for probabilities over object types

-   Rotate parameters model

    -   Use pre-trained word embeddings

    -   Word embedding size of 200

    -   LSTM size of 119

    -   Dropout of 0.11

    -   *sigmoid* activation for binary output (clockwise or anti-clockwise)

-   Push parameters model

    -   Do not use pre-trained word embeddings

    -   Word embedding size of 100

    -   LSTM size of 286

    -   No dropout

    -   *tanh* to obtain $x, y \in [-1, 1]$ push direction

-   Description model

    -   Trajectory encoding of 512

    -   Trajectory LSTM size of 32

    -   No encoder dropout

    -   Decoder LSTM size of 68

    -   Decoder dropout of 0.03

    -   Object encoding size of 512

    -   30 trajectory samples

    -   Action description encoding size of 512

    -   Action description LSTM size of 32

Scalability of physics engines
==============================

The proposed system incorporates a physics engine for physical prediction.  The input to this engine is a set of object poses and a set of action parameters, and the output is a set of trajectories for the objects.  Physics engines provide a number of advantages:

-   Principled numerical approximation of Newtonian physics. 

-   Natural input and output representations. 

-   Generalisation to arbitrary object shapes and properties. 

-   Zero learned parameters inside the engine. 

However, collision detection within these physics engines can be an expensive step, particularly for high-detail shape representations and complex geometries.  Consequently these engines can scale poorly with increasing scene complexity.  However, there are a number of techniques that can be employed to improve computational efficiency:

-   Complex geometries can be approximated by convex decompositions. 

-   Simple geometries can be approximated by collections of 3D primitives. 

-   The temporal resolution can be optimised to balance computational efficiency and accuracy. 

-   Computation can be parallelised and executed on a multi-core CPU or GPU. 

These techniques mean that the proposed system can be applied to and would scale to more complex scenarios and scenes without a prohibitive drop in performance. 

[^1]: These authors contributed equally to this work.