SEOs love Spiders…
By: Rob Hammond
Tags: Technical Seo, Crawling, SEO, Spiders

[caption id=”attachment_2137" align=”alignleft” width=”315"]

Photo credit: Rob van Hilten[/caption]

…and here’s a new, free spider for everyone.

One of the most common tasks in technical SEO is crawling websites. It becomes very difficult very quickly to effectively audit even a small site without some kind of automation, yet in our industry I’ve always thought there are too few tools available to do this.

While there are a number of enterprise-level SEO crawlers out there — DeepCrawl, 80legs for a more general crawler (BrightEdge and SearchMetrics also have their own relatively limited crawlers), these tools are expensive and not always hugely intuitive or suited to smaller, more focused tasks.

In terms of tools for SEOs confined by a small budget — or requiring more focused crawls, there are three main options I’ve used:

1. Xenu — great for finding broken links and duplicate titles, but is Windows only

2. Screaming Frog — easily one of the best small-scale SEO crawlers out there, with some good options available in the free version.

3. Integrity — basically Xenu for the Mac; good tool but not hugely useful beyond finding broken links.

My main gripe with all of those 3 tools is their client-side nature comes with inherent difficulties, especially for those in work environments with restrictive IT policies.

I’ve been hacking together web crawlers & scrapers for various tasks for almost as long as I’ve been doing SEO, but have never made any of them public or available in an interface outside of the command line.

After playing around with some of the new technologies available with HTML5, I’ve put together (I think) a pretty compelling web-based SEO crawler. It combines some of the best things about the tools above, and used properly it can be a pretty effective tool for helping put together SEO audits.

Some of its key features are:

Speed — it’s fast (albeit with low usage thus far…). Feedback is also real-time thanks to the HTML5 technology mentioned earlier. This does mean that it won’t work at all with Internet Explorer, but hey who uses that anyway.

— it’s fast (albeit with low usage thus far…). Feedback is also real-time thanks to the HTML5 technology mentioned earlier. This does mean that it won’t work at all with Internet Explorer, but hey who uses that anyway. Flexibility — with no options set it’ll just crawl, starting at the URL you enter into the box. If you’re after something a bit more advanced, the ‘advanced options’ let you customise the crawl so it’s laser targeted at the information you want to get.

— with no options set it’ll just crawl, starting at the URL you enter into the box. If you’re after something a bit more advanced, the ‘advanced options’ let you customise the crawl so it’s laser targeted at the information you want to get. Accessibility — provided you’re running a modern browser (not IE), you can run crawls from pretty much anywhere, and export to Excel-compatible TSV files at the end of the crawl.

The crawler obeys robots.txt rules and is limited to 250 URLs; however by using the regular expression fields in the advanced options, it will crawl only sub-folders, subsets of pages, or whatever you tell it to, making the limit fairly academic.

In terms of the information it gives you, it’s not a million miles away from Screaming Frog or Xenu, but some of the key things to be aware of (and some are only visible in the export) are:

Page titles & meta descriptions — dull but needed, once the data is in a spreadsheet it’s easy to find duplicates by sorting by these columns

— dull but needed, once the data is in a spreadsheet it’s easy to find duplicates by sorting by these columns Full redirect chains — does what it says on the tin…

— does what it says on the tin… Potential SEO errors — for example duplicated robots meta tags or mismatched rel=canonicals, can be easy to miss when you’re looking for other things

— for example duplicated robots meta tags or mismatched rel=canonicals, can be easy to miss when you’re looking for other things Link counts — external, external followed & total links

— external, external followed & total links Custom search parameter — you can use the ‘search pattern’ field to check for the presence of any given string of HTML or text within your pages — for example checking for correctly implemented

It’s not necessarily intended to be a replacement for a dedicated tool like Screaming Frog (partly because it’s been put together outside of my day job), but it can be a good complement or alternative in certain circumstances. It can also be used quite effectively with some of the other SEO tools I’ve released for deep-dives on specific issues.

If you’re interested in learning more/helping out, or have any feature requests, comment below or let me know directly!

Also if you like it and start using it feel free to drop me a link — my website sure could use it :)