DEF CON 17 - Michael Schrenk - Screen Scraper Tricks Extracting Data from Difficult Websites
Channel: DEFCONConference
Date: 2013-11-20

Screen Scraper Tricks: Extracting Data from Difficult Websites
Michael Schrenk

Screen scrapers and data mining bots often encounter problems when extracting data from modern websites. Obstacles like AJAX discourage many bot writers from completing screen scraping projects. The good news is that you can overcome most challenges if you learn a few tricks.

This session describes the (sometimes mind numbing) roadblocks that can come between you and your ability to apply a screen scraper to a website. You'll discover simple techniques for extracting data from websites that freely employ DHTML, AJAX, complex cookie management as well as other techniques. Additionally, you will also learn how "agencies" create large scale CAPTCHA solutions.

All the tools discussed in this talk are available for free, offer complete customization and run on multiple platforms.

Michael Schrenk is a webbot developer and the author of "Webbots, Spiders, and Screen Scrapers" (2007, No Starch Press). He has also written for ComputerWorld, php|architect and Web Techniques magazines. Mike also gave presentations at DEFCON X, XI and XV. He works for a wide range of clients across North American as well as in Russia, Spain and The Netherlands. Stop by www.schrenk.com and say hello.

For copies of the slides and additional materials please see the DEF CON 17 Archive here: https://defcon.org/html/links/dc-archives/dc-17-archive.html