BACKLOG-4885: Implement configuration delete