Before revision:  Specifically, on an input video sequence, STSGR (i) creates a two-stream visual and semantic scene graph on every frame, (ii) conducts intra-graph reasoning using node and edge convolutions generating visual memories, and (iii) applies inter-graph aggregation to capture their temporal evolutions. These visual memories are then combined with other modalities and the question embeddings using a novel semantics-controlled multi-head shuffled transformer, which then produces the answer recursively.
After revision:  Specifically, on an input video sequence, STSGR (i) creates a two-stream visual and semantic scene graph on every frame, and an inter-frame aggregation module capturing temporal cues.