Do not remove config that we want.

Signed-off-by: Philip Kuryloski <pkuryloski@pivotallabs.com>