Fixed vacancy classify to batch processing. Fixed robots.txt to prohibit access for robots