[core] Fix security issue where all users are considered admins of no admins specified in config.

This fixes issue #182