Why did infinite scroll fail at Etsy? - danwin.com

Words, photos, and code by Dan Nguyen. The 'g' is mostly silent.
Why did infinite scroll fail at Etsy?
My point is not that infinite scroll is stupid. It may be great on your website. But we should have done a better job of understanding the people using our website” – Dan McKinley, Principal Engineer at Etsy
A few weeks ago, Etsy engineer Dan McKinley gave a talk on “Design for Continuous Experimentation.” It’s an interesting, humorous presentation on large scale A/B testing and I fully recommend you check it out (slides and video here).
The moral was that A/B testing – much like the code it tests – must be done in a modularized fashion. The “fail” case he gave was when Etsy spent months developing and testing infinite scroll to their search listings, only to find that it had a negative impact on engagement.
McKinley said Etsy’s main lesson from this was that their A/B testing strategy was too monolithic:
Spend a ton of time building the infinite scroll feature and release it.
Verify that people love it, then find a Brooklyn warehouse to throw a celebration party.
“Seeing more items faster is presumed to be a better experience”, McKinley said. But the A/B tests showed various negative effects of the feature, including fewer clicks on the results and fewer items “favorited” from the infinite results page. And curiously, while users didn’t buy fewer items overall, “they just stopped using search to find these items.”
So basically infinite scroll failed in every major way. But not only was Etsy’s team wrong in assuming that users would benefit from infinite sroll, McKinley said, they were wrong in automatically accepting the two underlying assumptions behind infinite scroll:
Users want more results per page.
Users want faster results.
(Assumption #2 was tested by artificially slowing down search for a cohort of users. These users did not necessarily react positively to slower results. But their engagement level was not statistically significantly lessened.)
The point of McKinley’s talk was that instead of having the goal of “test infinite scroll,” Etsy realized it needed to test each assumption separately, and this going forward is their game plan (the success case McKinley gives is the revamp of Etsy’s search box).
Even if user engagement isn’t positively influenced by more results per page or by faster results, why does the combination of both have a decidedly negative impact?
The decision to try infinite scroll was partially influenced by Google apparent success with “instant search results,” according to McKinley. And it’s a feature that is prominent among popular Tumblr themes, Pinterest, and of course, at Facebook and Twitter, so presumably their A/B testing has yielded good results.
But why not at Etsy? Or at Amazon, which sticks with 16 results per search page? Users are notoriously fickle about interface changes. But if the search algorithm still brings up the best results at the top, then a user who has only 16 options before clicking through has no better advantage than the user who has 16 plus an “infinite” number of lesser results, if the latter doesn’t have to do any additional work to get them.
McKinley said he didn’t know why infinite scroll didn’t succeed for Etsy. There wasn’t, as far as they could tell, a technical fault (i.e. infinite scroll breaking in a specific browser). It was just a bad thing and the reason why wasn’t clear.
Now the actual merits of infinite scroll itself is still a controversial feature – even if there aren’t technical issues, which there almost always are. But in Etsy’s use case, it seems that at worst there should have been no effect, not negative effect. The most jarring problem of infinite scroll is that there’s no footer, which presumably the average Etsy user doesn’t need when making viewing/purchasing decisions.
Maybe it’s not that users consciously dislike infinite scroll. But in practice, maybe they lose a sense of orientation? Some users may have the habit of going further and further into search results – “playing the field” so to speak – before they realize that pages 1 and 2 are the best options they have. With pagination, it’s fairly easy to get back to those pages.
But if these users don’t have the habit of bookmarking/favoriting/writing-down-the-names of items as they scroll through, perhaps the deluge of infinite scroll makes it more likely for users to forget where they once were and what once caught their eye? Or maybe it’s a lack of developing the habits needed to actually act: when users are forced to deal with the loading time of each page, they learn to do their favoriting/click-throughs/impulse-purchasing on those pages before moving on? These users, when allowed to scroll for more items than they were used to, might simply lose the habit of click-to-favorite/browse/buy – but never develop the habit of scrolling back up. Or they’re just overwhelmed from the information overload and don’t feel like taking action any longer.
Infinite scroll may be pleasant for browsing, but does it lead to inaction? It’s an interface issue that is likely less a technical question than a psychological. I wonder what other online retail interfaces have found?
On Hacker News, user oconnore makes a great observation here, in that least some users, when returning to the search page after backing up from a product page, would find that they had lost their place in the infinite search stream (probably the biggest problem with infinite scroll implementations). I didn’t mention this in the original version of this post, but McKinley opened his talk with an anecdote about another poor assumption: power users (who worked at Etsy and made the suggestion) often opened up search results in new windows because they wanted to do side-by-side comparisons. But when Etsy made that the default behavior, testing found that most users did not appreciate it.
So putting 2 and 2 together: Perhaps many Etsy users have the habit of clicking-through a search result and then backing up to the search page. When the infinite-scroll didn’t properly mark their place, they *really* got lost and the search experience would obviously not be a good one. Seems as good as explanation as any, and so maybe it really is a technical issue after all.
Another update: HN user gfodor, who worked on the project, said the back-button problem may have applied to some IE users. But as McKinley says in his talk, the negative search effect occurred across all browsers, including ones that did back-up correctly. McKinley does go into good detail about how Etsy determines control group composition (sellers vs. non-sellers, for example, are a much different user type), but as this blog post is near-infinitely long I suggest again you check out his excellent talk on his blog.
176 thoughts on “Why did infinite scroll fail at Etsy?”
Is it possible that infinite scroll on sites like Amazon and Etsy didn’t work because it overloaded the user’s brain with choices?
If you’ve read much about choice architecture and psychology, you know that human brains are easily overwhelmed by too much choice. The consequences are lower satisfaction (“maybe I would have preferred option 17″) and increased cognitive processing to even evaluate the choices.
With pagination, the user can group a page’s results together. Then on the next page, they can compare the 16 options with the best of the previous page (or, finding what they want, stop looking). Infinite scrolling forces them to compare each new item with the entire past, rather than mentally archiving the previous page results.
I know that for shopping, I don’t like infinite scroll. It just feels… mentally draining. Not to mention there’s no clear start/stop or grouping inherent to the choices.
I agree with TRAVIS L, I don’t want an infinite scroll when shopping, since I would keep scrolling and my mind would become “overflowed” with choices. I rather would like filters where I can sort by price, price+shipping cost, reviews, purchases from different users, likes/upvotes, rating (do keep in mind 5 star rating generally suck, you need a decent algorithm that takes upvotes, downvotes, number of total votes etc in account), views/popularity..
Brilliant point Travis… I’ve always frowned on infinite scrolling but never knew why I didn’t like it. But I think you’re right that our brains just operate better with discrete buckets of information that facilitates comparing and evaluating. Of course, for some applications the comparing doesn’t need to happen so much (such as Twitter feeds and so on), but for search results that always relate back to the search criteria, it makes more sense to avoid the appearance of an endless list (even if the paginated lists are basically endless themselves.)
Travis is correct. It depends on the context of your design and how that content is delivered. As Marissa Mayer, when she was at Google explains, the content that is being displayed is a determining factor. Google Images uses infinite scroll because users are able to scan and process images much more quickly than text. Reading a search result takes much longer. This is the reason why their main search results still use the more traditional pagination technique.
That’s exactly what i was thinking when reading article. Shopping means that you must make a decision, make choice. And it’s much more easier to compare 16 items and then move on, rather that compare “infinite” number of items.
“Paradox of Choice” is exactly the effect I was wondering. Quite a few of the designers at Etsy came from Google, so I’m guessing that e.g. they were interested in applying their learnings that fast search = better experience. Really interesting to learn that it’s a different result.
An interesting additional component of UX are different search strategies on Google versus those on Etsy & other shopping sites. On Google you have a lot more “satisficing” searches â€” “show me the best information for my need and I’m done”. Etsy or Amazon are much more evaluative: the user is trying to find the single best solution to their problem, so the amount of choice becomes much more overwhelming more quickly. At Google this type of deep choosing / evaluation is a much smaller fraction of searches.
It makes me infinitely sad that the infinite scroll didn’t pan out for Etsy. I’m a huge fan of it. If users are getting lost, why not educate them on the change(so they are better about marking a favorite or opening a new tab with a particular item of interest)? Even better, why no create some feature within the infinite scroll that allows for temporary placeholding or marking on the page, so the user can quickly find it?
I think I desire this so much with Etsy because I find their search to be absolutely awful. I’m inundated with far too much product that I don’t care to see. I’ve spent wasted too many clicks navigating around trying to find items that remotely match what I’m looking for.
Infinite scrolling are good for contents spanning in social media, user would like just consume the content lazily. On a site like Etsy, Amazon user is browsing with some purpose, also “loading more” with a button click is an intutive User interface and there is a finite end. In the case of “infinite scroll” user wont know whats the end of the listing
I think e comparison with Pinterest, tumblr, fb and twitter is not appropriate, since all those are discovery user experiences, very different than search. In the social / feed sites users are consuming content in a more random way than on the shopping sites where users are usually looking for a specific product, therefore the ranking factor and 1st page content is that important.
I love this article. It’s an important emotion in the user experience. In my work, search is a must. It’s an order entry mechanism for 250 to 750 items. If they don’t have it in there order history, they must find it. Well, how? As I look at this problem, either as informed consumer, or a mere bowser..how can I display accurate search results? Dan mentioned that he wasn’t sure why it didn’t work. Why the infinite scroll was bad, but it was. It’s because there’s no suggestion. When we suggest an outcome in our searches, we get hits. If we don’t suggest and let them browse through 16,000 possible outcomes, we get no increased outcome (250-750 is a normal range on a customers line items, there are 16K+ in feasible procurement items). The power of suggestion is real. That’s why other algorithms work. I think, for Etsy, however this may be bad…
Nice article. Infinite scroll functions, I just don’t know the user-end case where I want/desire it. When I’m on a site or page with infinite scroll results, I immediately assume a dilution of quality results about three scrolls in. It’s more of a degraded dump than a quality results return.
Problems with infinite scroll in the context of Etsy is exactly what you’d expect to find when viewed through the lens of “scarce cognitive resources”. More choices = more resource drain. With Etsy, the choices have far greater significance than in something like Pinterest. There’s little downside to favoriting a picture… until it becomes part of a purchasing decision. When the choices *matter*, every additional choice is a costly resource hit to the brain. It’s not surprising that power users developed strategies for optimizing their cognitive resources within the Etsy context, but also not surprising that most others struggled.
The problems with choice (the “Parodox of Choice” is a good starting point) can be tough to uncover because they’re counter-intuitive for both developers and their users: we *think* that when it matters, MORE choice is better. Yet the opposite is usually true, though it’s largely a subconscious problem. One example of the choice-drain is in the typical US supermarket. The impulse, mostly sugary items are sitting right there at the checkout counter where they are much harder to resist. Not hard to resist *because you are hungrier*, but because you have just exhausted cognitive resources by doing the main activity of grocery shopping: making choices. By the time you reach the end, those scarce resources are drained, and , oh yes, we NOW know the same cognitive resources used in making choices and solving problems are ALSO those ones needed for self-control. And the sugary items are especially tempting because the one thing that will most quickly refill the choice-drained tank is glucose in the brain.
Infinite scroll is a fail in all mainstream websites because after scrolling down for a while you realize that you don’t know where you are anymore. If you refresh the page, the page is reset, and you cannot quickly go back to the same spot you spent time scrolling down to. With pagination you have a sense of navigation, why reinvent something that works already.
If it breaks the back button, I donâ€™t see why thereâ€™s any further pondering of this point required â€” though I will note that it probably also screws up Save Page… and Send Page… features in browsers that have them, and probably doesnâ€™t play well with bookmarking, either.
If I have been scrolling â€œinfinitely,â€ follow a link, and then hit Back to find myself back up at the top of the page â€” or part way down it, but not where I was â€” which of these things do I say?
â€œHyuk, okey-dokey, time to hit Page Down about a million times!â€
â€œOh, no, I am a Sad User and Totally Lost. Computers are hard.â€
â€œAh, you people are incompetent. Screw you and your broken page, which I shall now avoid.â€
I agree with Tom. To me this seems like a simple Search vs Discovery story. Clearly your testing has found that your Etsy users like to search for their stuff rather than endlessly browse.
We’ve been using infinite scroll for a while now on Home Ideas and our users appear to love it – we’re sure that’s because the site is pre-disposed towards browsing and browsing and browsing and browsing … for house porn.
A critical feature which helps our infinite scroll experience, and which avoids the problem of ‘not breaking the back button’ is that we open our items from the infinte scroll page into a lightbox. Once the user is done with that, the lightbox closes and the user resumes where they left off in the infinite scroll.
You mention ‘power users’ a couple of times and seem to give these users a lot of influence when deciding on features. But at the same time you also mention that these power users work for Etsy – they’re not power users at all, they work for you!
Your real users are the hoi polloi out there on the web, and they’re the ones you should be listening to.
I used to like the site Coolhunting but ever since they added infinite scroll I just can’t stand it. The site seems to load ahead almost logarithmically in chrome. I’d rather just click a button when I’m ready for more.
Sites like Google images, on the other hand, seem unable to keep up with me. I’d rather have a fixed amount of content load in the background while I’m doing something else. Then I can scroll through it insanely quickly, and send it off to fetch more while I go back to some other task.
Is there anyone who is happy with an infinite scroll implementation anywhere?
Interesting. I heard a similar story from Facebook awhile back when we asked why their website doesn’t have a fixed position header. They said that they had found people were less likely to click “Like” on things if they did it. Does that make any sense? No. Does it explain why the same users are happy with a fixed position header in native apps? No. I worry a bit that we live in a time where blindly following A/B Testing results is more important than actually understanding them.
I have no idea what Etsy’s implementation looked like, but another factor I’ve run into with infinite scroll on Google’s Image Search is that they actually scroll their search bar off screen. After scrolling down three or four pages worth of results, it makes it especially hard to go back and edit your search if you haven’t found what you want. I’m more likely to just give up and move on. Bing’s implementation is better in that case, and fixes the header at the top of the screen.
Refining Travis’ comment, infinite scroll would make sense when each item is evaluated individually, like Tumblr, rather than in comparison with all the other products displayed.
I suspect that the mental model the designers used was that there was a single “correct” choice that the user would recognize when they saw it, so it’s simply a matter of scrolling until it is found. (Or perhaps they hoped that each item is evaluated for purchase independently, leading to possible multiple purchases.)
However, it seems far more likely that the mental mechanism is to choose the best item among those displayed, in which case an infinite scroll means that there is no natural decision point (which loses a number of users), and for those who make their own decision point, there’s Travis’ point about the difficulty of recalling all the candidates and then re-finding a selected candidate that is remembered and chosen.
Two significant things about infinite scroll came to mind while I read this post, both based on our customer testing:
1) On mobile devices, while using a responsive design, infinite scroll can be a PITA because one loses the ability to access content that is shifted from a sidebar location (on desktop) to below the content area (on a mobile device). When reaching the end of the content container div, the infinite scroll “kicks-in” and causes the viewport to fill-up with more results. A virtual “race to the footer” takes place… not good for user interaction;
2) Even when using breadcrumbs, infinite scroll definitely removes the ability to drill-down and then dwell upon certain items. The “cost” of clicking on an item in infinite scroll is that you have to “go back to square one” when you are done, instead of moving horizontally or incrementally back a step or two, as you can with traditional content display.
Infinite scroll HAS worked really well for us when there is short, self-contained content that does not require any more details than what appears on the front page (such as embedded video thumbnails that expand to fill screen without losing one’s place on page one).
I thought the lightbox idea in the comments above sounded very promising!
To me, the most likely reason is the last one, that its harder to navigate back up the list to where you remember something good once you’ve decided you’re done searching. Maybe some dynamic anchor links that jump you back up through the results would enable users to stay navigated?
Could it also be that we’ve been “trained” to expect the best results in the first few pages of a search? So that often we completely disregard what shows beyond page 3 or so. Infinite scroll takes away the delineation between “I think these are relevant” and “I’m not even gonna bother with that other crap”. You just get everything, and like mentioned before, that’s exhausting and you’re not sure when to stop scrolling and start really evaluating.
Why did you remove the meta description from this page in the past few days?! Totally screwed my unit test! Why of all the articles I chose to unit test my meta description extraction, I chose this lame one?
I had the exact opposite takeaway. Infinite scroll didn’t fail. Sales didn’t suffer, but people stopped “hearting” things, which is a motivator for sellers to keep at it. It’s a non-monetary treat. The “fail” was not for the people shopping, the “fail” was for the Etsy system of non-monetary seller validation.
Maybe it was so successful and useful to browsers that they no longer felt they needed to ‘heart’ things to save them for later, since it was so easy to just find the damn thing again without paging through lists.
Commentors here brought up the difference between Google Images (infinte list) and Etsy, suggesting that “in shopping, you’re evaluating and comparing more,” which is a good point, except that Etsy’s results are just a big list of images…
Maybe Etsy should have added a ‘compare,’ or ‘curate’ button to the items, perhaps where “hearting” it would put a copy in a floating left-column overlay. If you could easily save that set as a Treasury, you’d have your ‘engagement’ back again.
In most cases I truly hate infinite scroll, because one cannot conveniently navigate, or bookmark the page. The one example, where infinite amazed me as a part of very good interface is Quartz website.
Infinite works best when there is extra navigation for content in it, and when it’s neatly connected with changing URLs.
The article touches on a key point then lets it slip through it’s fingers. Any new, cool widget cannot ignore basic needs. Infinite scroll can totally preserve scroll-to position on returning there from a deeper page, for example (it does NOT have to break the back button). If it doesn’t work well for all users, you did it wrong.
And therefore, about 2/3 of the times I have specified infinite scroll I have pulled back and made it a pagination control later on when development gets in a tizzy and cannot figure out how to do it right. I’ll take a clunky but functional control over a cool but broken one any day.
Without the details on the implementation, and with only analytics it’s hard to tell why Etsy failed. Or, if they could have done some minor tweak to fix their infinite scroll, vs. trashing it.
Same, same for any widget. The pattern is not invalid because someone implemented it wrong, or it was inappropriate for the situation. And without more info, I can’t say more about why it failed.
I hate infinite scroll with a passion, no matter how well it is implemented. In most cases I want to know how many records there are (assuming the page is made of discrete items such as search results) and I want to be able to navigate to any point. With infinite scroll I can’t navigate directly to the end or the middle or some other point without scrolling through a ton of content I’m not interested in.
Other people have already mentioned most of my other (numerous) objections, but no one has mentioned accessibility. Infinite scroll is a disaster for so many people using keyboard navigation, screen magnifiers, screen readers, voice recognition software etc.
If you really must have infinite scroll for some unfathomable reason, for heavens sake give us a button to turn it off and have proper pagination links (with the option to choose the number of items per page of course).
Everything smart was already said in the comments. So I’ll comment with my experience:
I remember my surprise when I returned to Etsy, after a long time, to discover he infinite scroll. After a few minutes I realized that I don’t want to continue searching anymore.
I guess I’m used to pagination, but not the UI aspect of it, but to the way I’m used to allocate my time for each search. I usually treat each page as a relevance filter, and know that after examining the 1st page results, I may browse through the second, but that’s all. I’ll make my decisions based on that. Infinite scroll interrupts my system
The fact that I’m totally ok with Google’s image search supports that point. When I look for an image on Google, relevance usually is not the issue. Variance is..
This situation speaks to a larger issue. So many companies now copy what other companies do because they like it or due to a “it’s working for them so let’s role it out too” mentality.
A more deliberate approach is to quickly prototype significant new approaches and put in front of people to see how they react/use it before allocating resources to build it out and a/b test it. Yep, talking about focus groups, triads or one-on-ones. If used for directional feedback they are invaluable. Of course, if used for literal feedback it’s just design by committee and nearly worthless.