Avoid tearing down process groups twice.

The second removal always triggers a warning.

Change-Id: Ieafa8374dbffdaff0d576de004a688cae6f8ac18