Lazily load IP addresses for bans