Added features

Added ability to ignore user agents and directories.