Face Search at Scale

Given the prevalence of social media websites, one challenge facing computer vision researchers is to devise methods to search for persons of interest among the billions of shared photos on these websites. Despite significant progress in face recognition, searching a large collection of unconstrained face images remains a difficult problem. To address this challenge, we propose a face search system which combines a fast search procedure, coupled with a state-of-the-art commercial off the shelf (COTS) matcher, in a cascaded framework. Given a probe face, we first filter the large gallery of photos to find the top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq1-2582166.gif"/></alternatives></inline-formula> most similar faces using features learned by a convolutional neural network. The <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq2-2582166.gif"/></alternatives></inline-formula> retrieved candidates are re-ranked by combining similarities based on deep features and those output by the COTS matcher. We evaluate the proposed face search system on a gallery containing <inline-formula><tex-math notation="LaTeX">$80$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq3-2582166.gif"/></alternatives></inline-formula> million web-downloaded face images. Experimental results demonstrate that while the deep features perform worse than the COTS matcher on a mugshot dataset (93.7 percent versus 98.6 percent TAR@FAR of 0.01 percent), fusing the deep features with the COTS matcher improves the overall performance (<inline-formula><tex-math notation="LaTeX">$99.5$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq4-2582166.gif"/></alternatives></inline-formula> percent TAR@FAR of 0.01 percent). This shows that the learned deep features provide complementary information over representations used in state-of-the-art face matchers. On the unconstrained face image benchmarks, the performance of the learned deep features is competitive with reported accuracies. LFW database: <inline-formula><tex-math notation="LaTeX">$98.20$ </tex-math><alternatives><inline-graphic xlink:href="wang-ieq5-2582166.gif"/></alternatives></inline-formula> percent accuracy under the standard protocol and <inline-formula><tex-math notation="LaTeX">$88.03$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq6-2582166.gif"/></alternatives></inline-formula> percent TAR@FAR of <inline-formula> <tex-math notation="LaTeX">$0.1$</tex-math><alternatives><inline-graphic xlink:href="wang-ieq7-2582166.gif"/> </alternatives></inline-formula> percent under the BLUFR protocol; IJB-A benchmark: <inline-formula> <tex-math notation="LaTeX">$51.0$</tex-math><alternatives><inline-graphic xlink:href="wang-ieq8-2582166.gif"/> </alternatives></inline-formula> percent TAR@FAR of <inline-formula><tex-math notation="LaTeX">$0.1$</tex-math> <alternatives><inline-graphic xlink:href="wang-ieq9-2582166.gif"/></alternatives></inline-formula> percent (verification), rank 1 retrieval of <inline-formula><tex-math notation="LaTeX">$82.2$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq10-2582166.gif"/></alternatives></inline-formula> percent (closed-set search), <inline-formula><tex-math notation="LaTeX">$61.5$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq11-2582166.gif"/></alternatives></inline-formula> percent FNIR@FAR of <inline-formula><tex-math notation="LaTeX">$1$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq12-2582166.gif"/></alternatives></inline-formula> percent (open-set search). The proposed face search system offers an excellent trade-off between accuracy and scalability on galleries with millions of images. Additionally, in a face search experiment involving photos of the Tsarnaev brothers, convicted of the Boston Marathon bombing, the proposed cascade face search system could find the younger brother's (Dzhokhar Tsarnaev) photo at rank <inline-formula><tex-math notation="LaTeX">$1$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq13-2582166.gif"/></alternatives></inline-formula> in <inline-formula> <tex-math notation="LaTeX">$1$</tex-math><alternatives><inline-graphic xlink:href="wang-ieq14-2582166.gif"/> </alternatives></inline-formula> second on a <inline-formula><tex-math notation="LaTeX">$5$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq15-2582166.gif"/></alternatives></inline-formula> M gallery and at rank <inline-formula><tex-math notation="LaTeX">$8$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq16-2582166.gif"/></alternatives></inline-formula> in <inline-formula> <tex-math notation="LaTeX">$7$</tex-math><alternatives><inline-graphic xlink:href="wang-ieq17-2582166.gif"/> </alternatives></inline-formula> seconds on an <inline-formula><tex-math notation="LaTeX">$80$</tex-math><alternatives> <inline-graphic xlink:href="wang-ieq18-2582166.gif"/></alternatives></inline-formula> M gallery.