Go version of Black Widow. Basic web app in Abacus, and basic fetcher/mysql store in fetch.go
Updating fetch with more logic
Latest stable - fetches a single page and stores. avoids dupes, stores host with ip, stores mime types for all requests but does not dl non-text files
Adding geo ip lookup function, but need to insert in to db still. adding quantcast top million sites text list for crawl seed
Stable fetcher and abacus web app with hosts/stats pages working; Need to implement search functionality and multi page crawler
Latest stable fetch and webapp - search function almost ready just needs to query sql and return results. fetcher stable needs crawler
Refactoring directories and files. Fetching is done in Go and accepts a link from STDIN. It inserts data to mysql, if there is a body it outputs it to STDOUT to go in to python extractLinks tool. Python extractLinks outputs parsed links to STDOUT for feeding back in to fetch program.
Cleaning up crawl script
moving fetch to widow folder
updating git ignore to ignore files that start with #
abacus web app now actually searches and returns some primitive results. No relationships or intelligence just straight keyword matching in SQL
Adding more robust search results with data structs and more useful info on header results. Added a new SQL reference query for joining hostname to headers. Updated search template
renaming a couple things and adding top 100 sites to be crawled
ran dos2unix on all files to fix bug and tweaked python script
Fixing abacus.go and search.html template to use sql.NullInt64 and sql.NullString types for to avoid null scan errors
Tweaking python script to run with python2 properly
Adding favicon and updated search template
Comment about empty header handling
Removing search by default if empty keywords. Hide sections on search page if fields are empty. Removing search through body fields
Merge branch 'master' of https://github.com/nanodano/BlackWidow