Cleanup these data structures. No need to keep blank stuff or index by keys.