Revert "Cache the sorted index of contacts"

This reverts commit 29356f2857c12eb8c63878d04851c27588393846.