Automatic Creative Selection with Cross-Modal Matching
Authors: name: Alex Kim, name: Jia Huang, name: Rob Monarch, name: Jerry Kwac, name: Anikesh Kamath, name: Parmeshwar Khurd, name: Kailash Thiyagarajan, name: Goodman Gu

Application developers advertise their Apps by creating product pages with
App images, and bidding on search terms. It is then crucial for App images to
be highly relevant with the search terms. Solutions to this problem require an
image-text matching model to predict the quality of the match between the
chosen image and the search terms. In this work, we present a novel approach to
matching an App image to search terms based on fine-tuning a pre-trained LXMERT
model. We show that compared to the CLIP model and a baseline using a
Transformer model for search terms, and a ResNet model for images, we
significantly improve the matching accuracy. We evaluate our approach using two
sets of labels: advertiser associated (image, search term) pairs for a given
application, and human ratings for the relevance between (image, search term)
pairs. Our approach achieves 0.96 AUC score for advertiser associated ground
truth, outperforming the transformer+ResNet baseline and the fine-tuned CLIP
model by 8% and 14%. For human labeled ground truth, our approach achieves 0.95
AUC score, outperforming the transformer+ResNet baseline and the fine-tuned
CLIP model by 16% and 17%.