How can I save a copy of the logs of my crawl for auditing them later?
It’s usually good to keep a copy of the logs of all your crawls to check for errors, exceptions, stats, etc. Pass a path of the file where you want the logs to be saved, in a dictionary to the cutom_settings parameter. A good practice for consistency is to give the same name to the output_file and log file (with a different extension) for easier retreival.