If you get a mortgage directly from a bank or credit union, you don’t need to use a mortgage broker. Some B lenders or private lenders, on the other hand, can only be accessed if you use a broker.