Avoid deletion of members from other groups

Signed-off-by: Justin Lecher <jlec@gentoo.org>