Revert "removal of ancestor indexing and dont-tokenize"

This reverts commit 0d61853e4cafc0134b1615cf56f1f4c1417ff22c.