Revert "Cache the sorted index of contacts"

This reverts commit 29356f2857c12eb8c63878d04851c27588393846.
This is causing crashes/regression, so it isn't worth it as a performance
improvement.
Merge pull request #40 from rburchell/master

Revert "Cache the sorted index of contacts"