Update dependencies
Use the File's method to check if the url is allowed to be indexed for a UA