Added function and end point for fetching all documents, merged in automated scraping every minute.