Delete removed config objects after SIGHUP

Fixes #3620