Correctly handle layer add/remove in modify interaction