If you were to decide to not pay your HOA fee then, depending on the agreement you and the HOA have made, the HOA has the right to charge late fees, or even take measures toward foreclosing on your property.