Make sure keys are actually unique even after reloading

Test Plan:
hack hack hack