[25] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,” arXiv preprint arXiv:2111.02114, 2021.
[26] S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman, GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, version 1.0, If you use this software, please cite it using these metadata., Mar. 2021. DOI: 10.5281/zenodo.5297715. [Online]. Available: https://doi.org/10.5281/ zenodo.5297715.
[27] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al., "Opt: Open pre-trained transformer language models," arXiv preprint arXiv:2205.01068, 2022.
[28] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., "Segment anything," arXiv preprint arXiv:2304.02643, 2023.
[29] W. Kim, B. Son, and I. Kim, "Vilt: Vision-and-language transformer without convolution or region supervision," in International Conference on Machine Learning, PMLR, 2021, pp. 5583-5594.
[30] J. Yang, J. Duan, S. Tran, Y. Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, “Vision-language pre-training with triple contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15671-15680.
[31] S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V. Vinay, and A. Grover, "Cyclip: Cyclic contrastive language-image pretraining," arXiv preprint arXiv:2205.14459, 2022.
[32] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, "Filip: Fine-grained interactive language-image pre-training," arXiv preprint arXiv:2111.07783, 2021.
[33] A. Fürst, E. Rumetshofer, V. Tran, H. Ramsauer, F. Tang, J. Lehner, D. Kreil, M. Kopp, G. Klambauer, A. Bitto-Nemling, et al., "Cloob: Modern hopfield networks with infoloob outperform clip," arXiv preprint arXiv:2110.11316, 2021.
[34] Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, "Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm," arXiv preprint arXiv:2110.05208, 2021.
[35] Y. Gao, J. Liu, Z. Xu, J. Zhang, K. Li, and C. Shen, "Pyramidclip: Hierarchical feature alignment for vision-language model pretraining," arXiv preprint arXiv:2204.14095, 2022.
[36] R. Herzig, A. Mendelson, L. Karlinsky, A. Arbelle, R. Feris, T. Darrell, and A. Globerson, Incorporating structured representations into pretrained vision & language models using scene graphs, 2023. arXiv: 2305.06343 [cs.CV].
[37] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in ECCV, 2020.
[38] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, "Visualbert: A simple and performant baseline for vision and language," ArXiv, vol. abs/1908.03557, 2019.
[39] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al., "Oscar: Object-semantics aligned pre-training for vision-language tasks," in European Conference on Computer Vision, 2020, pp. 121-137.
[40] H. H. Tan and M. Bansal, “Lxmert: Learning cross-modality encoder representations from transformers,” in EMNLP, 2019.
[41] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei, “Scene Graph Generation by Iterative Message Passing,” in Proc. CVPR, 2017, pp. 3097–3106.
[42] R. Herzig, M. Raboh, G. Chechik, J. Berant, and A. Globerson, "Mapping images to scene graphs with permutation-invariant structured prediction," in Advances in Neural Information Processing Systems (NIPS), 2018.
[43] R. Krishna, I. Chami, M. S. Bernstein, and L. Fei-Fei, “Referring relationships,” ECCV, 2018.
[44] A. Jerbi, R. Herzig, J. Berant, G. Chechik, and A. Globerson, “Learning object detection from captions via textual scene attributes,” ArXiv, vol. abs/2009.14558, 2020.
[45] M. Raboh, R. Herzig, G. Chechik, J. Berant, and A. Globerson, "Differentiable scene graphs," in WACV, 2020.
[46] F. Baradel, N. Neverova, C. Wolf, J. Mille, and G. Mori, “Object level visual reasoning in videos,” in Proc. ECCV, 2018, pp. 105–121.
[47] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al., “Relational inductive biases, deep learning, and graph networks,” arXiv preprint arXiv:1806.01261, 2018.
[48] F. Liu, G. E. T. Emerson, and N. Collier, “Visual spatial reasoning,” Transactions of the Association for Computational Linguistics, 2023.
[49] C. Gao, J. Xu, Y. Zou, and J.-B. Huang, “Drg: Dual relation graph for human-object interaction detection,” ArXiv, vol. abs/2008.11714, 2020.
[50] K. Kato, Y. Li, and A. Gupta, “Compositional learning for human object interaction,” in ECCV, 2018.