Rework automatic directory deletion

Check before deleting that we are allowed to delete
Ensure to not recursively delete directories not under prefix