Only remove the column if the column already exists

Fixes #759