# HalluciNet-_ing_ Spatiotemporal Representations Using a 2D-CNN
## 4 Experiments
### Using Multiple Frames to Hallucinate  
As previously discussed, there are situations (e.g., door open/close) where a single image cannot be reliably used for hallucination. However, motions cues coming from multiple frames can be used to resolve ambiguities.  
We modified the single frame HalluciNet architecture to accept multiple frames, as shown in Figure 4. We processed frame \(F_{j}\) and frame \(F_{j+k}\) (\(k>0\)) with our student 2D-CNN. In order to tease out low-level motion cues, we did ordered concatenation of the intermediate representations, corresponding to frames \(F_{j}\) and \(F_{j+k}\). The concatenated student representation in the 2-frame case is as follows:  
\[\phi_{s}=concat_{\phi}(\phi_{s}^{j},\phi_{s}^{j+k}) \tag{5}\]  
where \(\phi_{s}^{j}\) is the student representation from frame \(F_{l}\) as in Equation (2). This basic approach can be extended to more frames, as well as multi-scale cases. Hallucination loss remains as a single frame case (Equation (3).  
In order to see the effect of using multiple frames, we considered the following two cases:  
1. Single-frame baseline (HalluciNet(1f)). We set \(k=0\), which is equivalent to our standard single frame case;
2. Two-frame baseline (HalluciNet(2f)). We set \(k=3\), to give the student network \(f_{s}\) access to pixel changes in order to tease out low-level motion cues.  
We trained the networks for both the cases, using the exact same procedure and parameters as in the single frame case, and observed the hallucination loss, \(\mathcal{L}_{hallu}\), on the test set. We experimented with both kinds of actions--with short-term and long-term dynamics.  
Results for short-term actions are presented in Table 5 for UCF101. We saw a reduction in hallucination loss by a little more than 3%, which means that the hallucinated  
Figure 4: Multiframe architecture. Instead of using a single frame to hallucinate, representations of ordered frames are concatenated (\(\oplus\)), which is then used for hallucinating. Everything else remains the same as our single frame model.  
representations were closer to the true spatiotemporal representations. Similarly, there was a slight classification improvement, but with a 67% increase in computation time.  
The long-term action results are presented in Table 5(b) for MTL-AQA. Like with short-term actions, there was an improvement when using two frames. The percent of reduction in \(\mathcal{L}_{haltu}\) was better than the short-term case, and dive classification was improved across all components (except **AS**, which was saturated).  
Discussion:  
Despite the lower mean hallucination error in the short-term case, the reduction rate was larger for the long-term actions. We believe this is due to the inherent difficulty of the classification task. In UCF-101, action classes are more semantically distinguishable, which makes it easier (e.g., archery vs. applying_makeup) to hallucinate and reason about the immediate future from a single image, while in the MTL-AQA dive classification case, the action evolution can be confusing or tricky to predict from a single image. An example is trying to determine the direction of rotation--it is difficult to determine if it is forward or backward with a snapshot devoid of motion. Moreover, differences between dives are more subtle. The tasks of counting somersaults and twists need accuracy up to half a rotation. As a result, short-term hallucination is more difficult--it is difficult to determine if it is a full or half rotation. While the the two-frame HalluciNet can extract some low-level motion cues to resolve ambiguity, the impact is tempered in UCF-101, which has less motion dependence. Consequently, there is comparatively more improvement in MTL-AQA, where motion (e.g., speed of rotation to distinguish between full/half rotation) is more meaningful to the classification task.