Make them WeakHashMaps.

In case a player gets kicks, crashes or for whatever reason doesn't pass through the PlayerQuitEvent it'll remove their entry.