spec main crawler; finishing touches
better assertion on links count