Always a TreeMap, and document a bit.
singleKeyListeners is null for inserts, a TreeMap for SSKs (not salted), and a HashMap for CHKs (salted)