Fix EntityNotFoundException when removing active rules
(cherry picked from commit 1cf660f)