Apache Nutch (nutch)
Category: web-framework
Programming language: Java
Apache Nutch is a highly extensible and scalable open source web crawler software project. Stemming from Apache Lucene, Nutch is a well matured, production ready batch crawler relying on Apache Hadoop data structures, which are great for batch processing. Nutch has a modular architecture and provides pluggable and extensible interfaces such as Parse, Index and ScoringFilter's for custom implementations e.g. Apache Tika for parsing. Additionally, pluggable indexers exists for Apache Solr, Elastic Search, etc. Nutch can run on a single machine, but gains a lot of its strength from running in a Hadoop cluster. The now retired branch Nutch 2.x differed from 1.x in one key area: storage is abstracted away from any specific underlying data store by using Apache Gora for handling object to persistent mappings and to store fetch time, status, content, parsed text, outlinks, inlinks, etc. into a number of NoSQL storage solutions.
Maintainer: Nutch PMC
Release: 1.6 (2012-12-06)