But your current implementation won't work properly when there is more than 10 contacts that begin with D.

I just had a new idea. What do you think of the following algorithm:
1. if we have only 1 term, set filterLimit to 10.
2. take the term whose length is the greatest. If that length is < 3, set filterLimit to 10.
3. use the API with this term
4. do the filtering for the other terms (if there are other terms) in the app, without using the Contacts API

Step 2 should prevent having too many results and thus we'll stay performant. And we match other platforms.