Idan/fga-video-retrieval

Video-text matching on YouCook2

Built on the attention layer of Factor Graph Attention (CVPR 2019), following Video and Text Matching with Conditioned Embeddings (WACV 2022). A caption and a video are attended jointly and scored against each other, trained with a max-margin contrastive loss against the hardest negative in the batch.

Code: github.com/idansc/fga.

Read the split before the numbers

This is not YouCook2's official train/val division, and these numbers do not compare to published YouCook2 retrieval figures.

The feature dump available here covers YouCook2's validation videos only -- 436 of them -- so the data was divided over those by video, no video appearing on two sides:

	videos	captioned segments
train	296	2,354
val	52	379
test	88	617

Retrieval over 617 segments is an easier problem than retrieval over the official split's several thousand, and the numbers below should be read as an internal comparison only.

Results

Held-out test slice, 617 segments. Recall at k in both directions:

	R@1	R@5	R@10	median rank
text to video	0.418	0.723	0.818	2
video to text	0.476	0.786	0.882	2

The checkpoint is the one that scored best on the validation videos, not on the test slice. Selecting on the test slice reads 0.438 / 0.468 instead; the validation split exists so that choice is not made there.

R@10 here counts a hit at rank 10 or better. Worth stating because the reference implementation's evaluation.py computes its r10 as ranks < 50 in all four of its ranking functions -- a commented-out print in the same file labels that value "Ranks@50" -- so an R@10 reported by that code is an R@50 and is not comparable to the column above.

Training

Hidden size 768, learning rate 2e-4, 2,000 epochs. Video features are the InternVideo MM-L14 dump; captions are encoded with GloVe 300d.

Usage

from fga.tasks.video_retrieval import VideoMatchModel

model = VideoMatchModel.from_pretrained("Idan/fga-video-retrieval")
scores = model(caption_input_ids=text, video_features=video).logits

Citation

@inproceedings{ali2022video,
  title={Video and Text Matching with Conditioned Embeddings},
  author={Ali, Ameen and Schwartz, Idan and Hazan, Tamir and Wolf, Lior},
  booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision},
  year={2022}
}

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@inproceedings{zhou2018towards,
  title={Towards Automatic Learning of Procedures from Web Instructional Videos},
  author={Zhou, Luowei and Xu, Chenliang and Corso, Jason J},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2018}
}