Add the crawl_uri_filters list in the site config to allow you to filter out certain URIs during crawl.
Add expiry control from the site configuration.
Add a test that the TTL is obtained from config and make get_site_sets non-private to the module.