The IRS taxes unemployment at the federal level, and in some cases, the state level. However, you will not pay Social Security or Medicare taxes on those payments.