While it's rare, some contacts may be saved directly to your iPhone's memory rather than to one of your external accounts. If you follow the steps above to remove all contacts, the remaining ones are saved to your phone's storage. Unfortunately, there isn't a quick way to delete contacts stored on your iPhone's memory. You will need to delete them individually by tapping on Edit and then Delete at the bottom.