Fix concurrency issue in lifetime manager.

Change-Id: Id06fdad7e5ae4d4aec34b688e1be9548845cda2d