Move paths to remove to a set