updates for comment show logic
and robot.txt mapping