Borrowing from a 403(b) plan should not affect your credit score, as there's no credit check required and these loans are not reported to credit bureaus.