documentation for crawler files