I think we could benefit here from the following optimization:

To cache the results from previous queries i.e

Let's suppose with 3 digits we get a collection of 10 contacts, then the user adds a new number and we get a collection with only 4 contacts. if the user presses the backspace button we could immediately show the 10 contacts, because he is only going backwards to a previous query. 

With that optimization we could improve the performance of the 'backspace' button, avoiding new unneeded calls to mozContacts