As part of the refinancing process, you can take the points off as well. It is necessary to deduct refinancing points over the lives of the loans. A seller or a borrower who pays off the loan is allowed to deduct all of the points not deducted already (unless they refinance with the same lender).