nocko/Cornell-Spider-for-Unix — pull request #2 by NJnoc (2012-07-31 15:45:19+00:00)
Title: Adding multi-processing, Line scans and detailed logging.

Updated the files with autocrlf.

Added the following features:

The engine is capable of running block scans and line scans.
Block scans are for faster scans and are done on 64K blocks of the file.
Line scans are for more detailed scans and return the exact line number of the suspicious content. This is CPU intensive and slow but accurate and pinpoint the exact line number of suspicious content.
The engine checks for the number of cores on the machine one which it will run and fork into (cores-1) number of processes making the scans faster.
Added a simpler Luhn's algorithm to validate credit cards.
Added a new bash script that will can scan NFS mounts from a list of servers.

Details:
spider.conf has the following new directives:

dolinescan: 0 - disable line scan, 1 - enable line scan
hostip: IP of the host to be scan. Used when remote systems are scanned using NFS mounts.

scan.sh:

This is a script to scan remote systems based on the NFS mounts they export.
It takes a text file with a list of IPs to scan.
Creates log files for every IP being scanned.
Cleans up the mounts after the scan.

Please review and push.

Thanks,
Steve William