You can take the risk if you wish, as the Spanish tax authorities may not actively chase this tax.  However, there will most likely be problems when the property is to be sold.  We recommend that all non-residents pay this tax every year.