No. The USDA Annual Fee remains in place for the life of the loan. But you could refinance later into a conventional loan when you have 20% equity, to eliminate the fee.