per pull request review, conditionalizing more of the list removal code so that the working set list isn't changed at all
Merge remote-tracking branch 'origin/bchin/issue_4238' into bchin/issue_4238