Revert "Cache the sorted index of contacts"

This reverts commit 29356f2857c12eb8c63878d04851c27588393846.
This is causing crashes/regression, so it isn't worth it as a performance
improvement.