“Never Forget a Coupon”: How SnipSnap Used Visual Search to Crowdsource Billions in Savings
By: Ted Mann
Tags: Retail Technology, Coupons, Retail, Snipsnap, Apps

My adventures with visual search — and entrepreneurship in general—all started with a 20%-off buybuyBaby coupon. Or rather, me forgetting the buybuyBaby coupon while purchasing a $600 stroller, and then getting yelled at by my wife for being such an incompetent consumer.

Coupons are a powerful staple of our retail economy. $500 billion worth of coupons were distributed last year. 80% of Americans used them ever week. But for people like me, they are also a huge pain in the butt to remember. So I turned to my trusty iPhone, which I NEVER forget, and came up with a life hack: I started taking photos of every single coupon that arrived in the mail. As I went into stores and showed the coupon on my phone screen, most of the store associates seemed puzzled that I was trying to scan a picture, but they were willing (or coaxed) into zapping the barcode. And most of the time it worked. Even if they barcode didn’t register (as was often the case on those old laser scanners), they would key in the promo code.

Turning the “Bowl of Shame” into the SnipSnap Coupon App

Before long I’d become a half-decent couponer, and was even out-saving my wife. Who’s the “incompetent consumer” now?! Rather than leaving the physical coupons collect dust in a bowl in our kitchen—“The bowl of shame” — I’d snap a pic, then toss the paper in the trash, knowing that I had a digital copy on me at all times. Eventually my wife got in on it. And there was a viral coefficient, too, with folks standing behind me in line who would ape the practice, and even sometimes borrow my phone to scan the deal for themselves.

So, where does visual search fit into all of this? Well, just as I’d come up with a way to solve my forgotten-coupon problem, other issues had emerged—my camera roll was now clogged with coupon photos instead of baby pictures; I had so many saved, that I’d often forget what I had; and more and more people were asking me to share the coupons via text or email. This is what gave birth to the idea of SnipSnap.

The app took that use case of snapping coupons, and built an entire coupon experience around it. Users had the option to share certain coupons (manufacturer ones and single-use offers weren’t sharable), which went into a crowd-sourced coupon database. At any given time, the app had a million or so offers! But to manage this scale, and deliver things like expiration-date reminders and in-store coupon reminders (“Hey, looks like you’re at Bed Bath & Beyond. Don’t forget you have a 20% of coupon!), we needed to extract structured data from the coupons. It wasn’t enough to simply aggregate a bajillion coupon photos, we needed metadata.

Business-card scanning apps were quite good at the time (still are—future blog post coming!), and what SnipSnap needed, it seemed, was a way to automatically transcribe all the data from a coupon and populate it into the right fields: Face value, expiration date, store or brand, etc. However, while Optical Character Recognition technology was perfectly suited to doing this for business cards, it proved to be a fool’s errand to do it on coupons. The data simply wasn’t consistently formatted, and often the graphics to show things like face value and store weren’t even picked up by the software that companies like Abbey and Nuance had created. Enter visual search.

We quickly realized that some of the most basic scanning techniques, like barcode recognition, yielded immediate utility. We were able to match two of the same coupons that were scanned, and instantly return the correct information for both. Even more important, by recognizing the barcode on the paper coupon, SnipSnap could render a “clean” version that could be displayed on the phone screen at full brightness, making scanning in store possible.

We were also able to build straightforward models to recognize if a photo submitted to the app was a coupon or not a coupon. And yes, there were LOTS of photos of things that were clearly not coupons — everything from pets to feet to, uh, unmentionables. But the most challenging task for us was to recognize the key coupon data, like the retailer or the face value. This required us to build classifiers that were built on large number of coupon images—which we didn’t have at the outset. So we did what so many other startups who haven’t had the data or automation they aspire to on Day 1: We used people to manually transcribe these key details.

Location-based coupon reminders

Specifically, we used Amazon’s Mechanical Turk. There have been some great articles published on how we did this on both Fast Company on Amazon’s MTurk blog. In short, we came up with a very scalable way to transcribe massive quantities of coupons with an efficient form that MTurk workers in the US would fill out (we restricted to US only when we saw the accuracy go up substantially and the need for consensus reviews go down). This enabled us to return to SnipSnap users all of the key coupon details, and deliver on the promise of turning paper coupons into mobile ones. It also enabled us to power features like expiration-date reminders, location-based reminders, and a searchable coupon database of all shared offers.

Once we had millions of coupons tagged and annotated through the MTurk apparatus, we set to work taking this data and creating classifiers to automatically detect things like store logos. All of which has enabled SnipSnap over time to automate more and more of the coupon transcription process. We still rely on a human-in-the-loop fallback to identify coupons that can’t be automatically processed, but the number of offers handled this way has fallen substantially over time.

When SnipSnap launched, it was selected to be the top “New and Noteworthy” app in the Apple App Store, and grew to 200,000 users within a month. By the time we sold the SnipSnap app in 2015, the user-base had grown to more than 6 million! It’s still one of the most popular coupon apps, and certainly one of the go-to apps for scanning, finding, and redeeming in-store coupons. And it remains one of the most compelling cases for how visual search can be leveraged to solve a retail problem (the forgotten in-store coupon) and empower millions of shoppers.

Check it out!

Current App Store Listing for SnipSnap

SnipSnap for iOS

SnipSnap for Android

SnipSnap website