added local saved_urls to cache until after post_batch / s3 save
a bit wasteful for memory if we are saving already to s3_cache
could use memory optimization here