Simple Summary. Every year, your refund is calculated as the amount withheld for federal income tax, minus your total federal income tax for the year.