Created Abstract class for crawler updated to support Robots filter rule.
Added Robots Filter Rule
Added new Examples of parallel usage and moved out of src directory