Only prefix keys if we have a prefix