Interest continues to accrue on the diminishing unpaid balance of tax debt until it is paid off. Interest on unpaid debt and penalties for late or non-payment can add up to as much as 10% per year.