The IRS does offer a first-time penalty abatement waiver for some taxpayers that can eliminate accrued penalties as well as interest on the penalties. However, the interest on the original unpaid tax amount will continue to accrue. Providing the required documentation to request a first-time penalty abatement is best done with the help of an experienced tax professional.