Check if the URL is a directory before attempting to scan it