You, the business owner submitting your unpaid invoices for immediate payment, will be the one paying the factoring fee. You pay this fee when the factoring company withholds the discount rate (or factor fee) from the final remainder payment after your customer pays their invoice to the factoring company.