Home Action Genome Challenge:
Task 2
Team: AIST&DENSO

Kensho Hara,AIST

Yasufumi Kawano,

Yoshiki Nagasaki,

Yue Qiu,

Hirokatsu Kataoka,AIST, Keio University

AIST, Keio University

AISTAIST

Overall Framework:◆ The overall framework consists of image module, bounding box module, semantic module, and videomodule.
◆ The logits of each module are added together for object and predicate prediction.
◆ The network outputs scores for each object and predicate.
◆ The overall framework is inspired by RelDN [Zhang et,al., CVPR2019 ]Ablation Experiments on Different Modules:◆ Image module is important for obtaining better object accuracy for the current network.
◆ Video module tends to perform better for predicate prediction.

Ablation Experiments on Parameters: (without Video Module)◆ Details of Image module:
Resnet module + 2-layered MLP (input dimension -> hidden dimenson -> out dimension )◆ ResNet 152 and ResNet 101 are slightly better than ResNet 50.

Predicate prediction:◆We compute the distribution of predicate list for each object;◆We determine the score of each predicate list for all objects (prior scores);◆We record the scores of each predicate for all objects (predicted scores), whichis computed through the network;◆The final score of each predicate list for each object is computed by multiplying
the prior scores with the predicted scores.

Final results:Home Action Genome
Our submission:◆ Image-only module;◆ Image resolution: 224;
◆ Image feature extraction: ResNet 101;
◆ Object and Predicate prediction: 2-layered MLP with hidden dimension of 256.

Summary:◆ Image module is important for obtaining better accuracy for the current module.
◆ Video module tends to perform better for predicate prediction.
◆ The network needs to be improved for combining image and video information.

References:◆ Zhang, Ji, et al. "Graphical contrastive losses for scene graph parsing." Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition. 2019.
◆ Hara, Kensho, Hirokatsu Kataoka, and Yutaka Satoh. "Can spatiotemporal 3d cnns retrace the history of 2d cnns andimagenet?." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018.
◆ He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computervision and pattern recognition. 2016.

◆ Rai, Nishant, et al. "Home Action Genome: Cooperative Compositional Action Understanding." arXiv preprint
arXiv:2105.05226 (2021).
◆ Ji, Jingwei, et al. "Action genome: Actions as compositions of spatio-temporal scene graphs." Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
◆ Bertasius, Gedas, Heng Wang, and Lorenzo Torresani. "Is Space-Time Attention All You Need for Video
Understanding?." arXiv preprint arXiv:2102.05095 (2021).

References:◆ Zhang, Ji, et al. "Graphical contrastive losses for scene graph parsing." Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
◆ Hara, Kensho, Hirokatsu Kataoka, and Yutaka Satoh. "Can spatiotemporal 3d cnns retrace thehistory of 2d cnns and imagenet?." Proceedings of the IEEE conference on Computer Vision andPattern Recognition. 2018.
◆ He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE
conference on computer vision and pattern recognition. 2016.
◆ Rai, Nishant, et al. "Home Action Genome: Cooperative Compositional Action
Understanding." arXiv preprint arXiv:2105.05226 (2021).
◆ Ji, Jingwei, et al. "Action genome: Actions as compositions of spatio-temporal scenegraphs." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
2020.
◆ Bertasius, Gedas, Heng Wang, and Lorenzo Torresani. "Is Space-Time Attention All You Need forVideo Understanding?." arXiv preprint arXiv:2102.05095 (2021).