Even if you’ve covered Congress for The New York Times for a decade, it can be hard to recognize which member you’ve just spoken with.  There are 535 members, and with special elections every few months, members cycle in and out relatively frequently.  So when former Congressional Correspondent Jennifer Steinhauer tweeted “Shazam, but for House members faces” in early 2017, The Times’s Interactive News team jumped on the idea. 

Our first thought was: Nope, it’s too hard!  Computer vision and face recognition are legitimately difficult computer science problems.  Even a prototype would involve training a model on the faces of every member of Congress, and just getting the photographs to train with would be an undertaking. 

But we did some Googling and found the Amazon Rekognition API.  This service has a “RecognizeCelebrity” endpoint that happens to include every member of Congress as well as several members of the Executive branch.  Problem solved!  Now we’re just talking about knitting together a few APIs. 

As we began working on this application, we understood that there are numerous, valid privacy concerns about using face recognition technology.  And Interactive News, a programming team embedded in the newsroom, abides by all aspects of The Times’s ethical journalism handbook including how we gather information about the people we cover.  In this case, we decided that the use of Rekognition’s celebrity endpoint meant we would only recognize congress people and other “celebrities” in Rekognition’s database. 

By the end of the summer, Interactive News interns Gautam Hathi and Sherman Hewitt had built a prototype based on some conversations with me and my colleague Rachel Shorey.  To use the prototype, a congressional reporter could snap a picture of a congress member, text it to a our app, and get back an annotated version of the photograph identifying any members and giving a confidence score. 

However, we discovered a new round of difficulties.  Rekognition incorrectly identified some members of Congress as similar-looking celebrities — like one particularly funny instance where it confused Bill Nelson with Bill Paxton.  Additionally, our hit rate on photographs was very low because the halls of the Capitol are poorly lit and the photographs we took for testing were consistently marred by shadow and blur.  Bad connectivity in the basement of the Capitol made sending and receiving an MMS slow and error-prone during our testing.  And, of course, there were few places in the Capitol where we could really get the photographs we needed without committing a foul. 

Gautam and Sherman got around the “wrong celebrity” problem with a novel approach: A hardcoded list of Congresspeople and their celebrity doppelgangers.  We grew more confident taking photographs and only sent the ones where members were better lit. 

A text-based interface is easiest for reporters to use, so while texting is slow, it’s superior to a web service in the low-bandwidth environment of the Capitol. 

In addition to confirming the identity of a member, Who The Hill has helped The Times tell some stories we couldn’t have reported otherwise.  Most recently, Rachel Shorey found members of Congress at an event hosted by a SuperPAC by trawling through images found on social media and finding matches. 

If you’re interested in running your own version, the code for Who The Hill is open sourced under the Apache 2.0 license.  The latest version includes a command-line interface in case you’d like to use the power of Amazon’s Rekognition to dig through a collection of photographs on your local machine without sending a pile of MMS messages. 

Our service is far from infallible.  But Times reporters like Thomas Kaplan love having a backup for when they can’t get a moment with a member to confirm their identity. “ Of course,” says Kaplan, “the most reliable way to figure out a member’s identity is the old-fashioned way: Just ask them.”