changed task for checking galaxy so that it searches for robots.txt instead of needing to make a database call