remove links
add robots txt