It’s difficult, but not impossible, to sue the IRS. The taxpayer must show the IRS action to have been frivolous, malicious, or wantonly groundless. This means something more than that the IRS “guessed wrong” in determining the correct action.