Yea, I think that is a separate issue.  I would guess its the time it takes for the contacts API to return the null response indicating that there are no contacts.  Unfortunately it has to initialize all its machinery in order to determine that, I think.