adding code for crawler

as is
fixing security stuff