Yes. If you dont; you are giving the Government the chance to place a Lien against your Property. You will either have to relocate and rebuild the building/property; demolish the building/property and solve the Setbacks Violation Notice.