Every year, your refund is calculated as the amount withheld for federal income tax, minus your total federal income tax for the year. A large portion of the money being withheld from each of your paychecks does not actually go toward federal income tax.