Fix #82, fix #81: Workarounds for NPEs

Still no idea what's causing these, logging more data.
Switching to hashsets internally because I've been meaning to and why
not?