It's a drop at a time. It doesn't look bad until the bucket is full. Grandfathering, protected heavily under state law, and nonconformity are the biggest threats to preservation. One possible way to address the concerns is by amending the bylaw to set specific thresholds as to how much expansion of nonconforming lots is allowed. But even that is likely to raise the hackles of many property owners.
Any number chosen is subject to debate.