<PAGE>   1
                                                                 Exhibit 10.32.1

                               THIRD AMENDMENT TO
                    WAREHOUSING CREDIT AND SECURITY AGREEMENT

     This Third Amendment to Warehousing Credit and Security Agreement (this
"Amendment"), is entered into as of the 28th day of March, 2000, by and between
HANOVER CAPITAL MORTGAGE HOLDINGS, INC., a Maryland corporation ("Company"),
HANOVER CAPITAL PARTNERS, LTD., a New York corporation ("HCP")(the Company and
HCP herein collectively called "Original Borrowers"), and HANOVER QRS-1 98-B,
INC., a Delaware corporation ("QRS-1"), HANOVER QRS-2 98-B, INC., a Delaware
corporation ("QRS-2"), HANOVER SPC-A, INC., a Delaware corporation ("SPC"), and
HANOVER CAPITAL REPO CORP., a Delaware corporation ("Repo")(QRS-1, QRS-2, SPC,
and Repo herein collectively referred to as the "New Borrowers", and Original
Borrowers and New Borrowers herein collectively referred to as the "Borrowers");
and BANK UNITED, a federal savings bank ("Lender"). Capitalized terms used but
not defined herein have the meanings assigned to them in that certain
Warehousing Credit and Security Agreement (Single-Family Mortgage Loans) (the
"Credit Agreement") dated effective as of April 30, 1999, by and between
Original Borrowers and Lender, as the same has been or may be amended or
supplemented from time to time.

     Section 1. RECITALS. The Borrowers and Lender desire to amend the Credit
Agreement, subject to the terms and conditions of this Amendment. Therefore, The
Company and Lender hereby agree as follows, intending to be legally bound:

     Section 2. AMENDMENTS. The Credit Agreement is hereby amended and
supplemented as follows:

     (a) Section 1.1 of the Credit Agreement is hereby amended by the amendment
     or addition of the following definitions:

          "TERMINATION DATE" shall mean May 31, 2000, or such earlier date upon
          which Lender's obligation to fund shall be terminated pursuant to the
          terms of this Agreement.

     (b) Section 2.5(a) of the Credit Agreement is deleted in its entirety, and
     the following is substituted therefor:

               "(a) The outstanding unpaid principal amount of all advances
          shall be payable in full upon May 31, 2000."

     (c) The promissory note ("Credit Note") dated as of March 28, 2000, in the
original principal amount of $50,000,000, executed by the Borrowers and payable
to the order of Lender, is given to Lender in replacement of the promissory note
dated May 12, 1999, in the original principal amount of $50,000,000, executed by
the Borrowers and payable to the order of Lender (the "5/99 Note"), which 5/99
Note was given to Lender in replacement of the promissory note dated April 30,
1999, in the original principal amount of $50,000,000, executed by the Original
Borrowers and payable to the order of Lender (the "Original Note"), and not in
novation or discharge thereof. The



<PAGE>   2


definition of the term "Note" in the Credit Agreement is hereby amended to mean
the Credit Note and all renewals, extensions, modifications, increases,
rearrangements, and replacements thereof.

     Section 3. EXTENSION FEE. In consideration for Lender's agreement to extend
the term of the Credit Agreement, and as a condition precedent to Lender's
agreement to enter into this Third Amendment, Borrowers agree to pay to Lender
an extension fee equal to 0.25% of the Commitment, prorated for two months
($20,833.33).

     Section 4. REPRESENTATIONS. The Borrowers represent and warrant that all of
the representations and warranties contained in the Credit Agreement and all
instruments and documents executed pursuant thereto or contemplated thereby are
true and correct in all material respects on and as of this date.

     Section 5. CONTINUED FORCE AND EFFECT. Except as specifically amended
herein, all of the terms and conditions of the Credit Agreement and all other
Loan Documents are and remain in full force and effect in accordance with their
respective terms. All of the terms used herein have the same meanings as set out
in the Credit Agreement, unless amended hereby or unless the context clearly
requires otherwise. References in the Credit Agreement to the "Agreement," the
"Loan Agreement," "hereof," "herein" and words of similar import shall be deemed
to be references to the Credit Agreement as amended hereby. Any reference in the
other Loan Documents to the "Agreement," the "Line of Credit Agreement,"
"Warehouse Agreement," or the "Loan Agreement" shall be deemed to be references
to the Credit Agreement as amended through the date hereof. Any references in
the Credit Agreement or any of the Loan Documents to the Note, or the Credit
Note shall be deemed to be references to the Credit Note.

     Section 6. REPRESENTATIONS AND RELEASE OF CLAIMS. Except as otherwise
specified herein, the terms and provisions hereof shall in no manner impair,
limit, restrict or otherwise affect the obligations of the Borrowers or any
third party to Lender, as evidenced by the Loan Documents. The Borrowers hereby
acknowledge, agree, and represent that (i) the Borrowers are indebted to Lender
pursuant to the terms of the Credit Note; (ii) the liens, security interests and
assignments created and evidenced by the Loan Documents are, respectively,
first, prior, valid and subsisting liens, security interests and assignments
against the Collateral and secure all indebtedness and obligations of the
Borrowers to Lender under the Credit Note, the Credit Agreement, all other Loan
Documents, as modified herein; (iii) there are no claims or offsets against, or
defenses or counterclaims to, the terms or provisions of the Loan Documents, and
the other obligations created or evidenced by the Loan Documents; (iv) the
Borrowers have no claims, offsets, defenses or counterclaims arising from any of
the Lender's acts or omissions with respect to the Loan Documents, or the
Lender's performance under the Loan Documents; (v) the representations and
warranties contained in the Loan Documents are true and correct representations
and warranties of the Borrowers, as of the date hereof; (vi) the Borrowers
promise to pay to the order of Lender the indebtedness evidenced by the Credit
Note according to the terms thereof; and (vii) the Borrowers are not in default
and no event has occurred which, with the passage of time, giving of notice, or
both, would constitute a default by the any of the Borrowers of such Borrower's
obligations under the terms and provisions of the Loan


                                                                               2

<PAGE>   3

Documents. In consideration of the modification of certain provisions of the
Loan Documents, all as herein provided, and the other benefits received by the
Borrowers hereunder, the Borrowers hereby RELEASE, RELINQUISH and forever
DISCHARGE Lender, its predecessors, successors, assigns, shareholders,
principals, parents, subsidiaries, agents, officers, directors, employees,
attorneys and representatives (collectively, the "Lender Released Parties"), of
and from any and all claims, demands, actions and causes of action of any and
every kind or character, whether known or unknown, present or future, which the
Borrowers have, or may have against Lender Released Parties, arising out of or
with respect to any and all transactions relating to the Credit Agreement, the
Original Note, the Credit Note, and the other Loan Documents occurring prior to
the date hereof, including any other loss, expense and/or detriment, of any kind
or character, growing out of or in any way connected with or in any way
resulting from the acts, actions or omissions of the Lender Released Parties,
and including any loss, cost or damage in connection with any breach of
fiduciary duty, breach of any duty of fair dealing, breach of competence, breach
of funding commitment, undue influence, duress, economic coercion, conflict of
interest, negligence, bad faith, malpractice, violations of the Racketeer
Influence and Corrupt Organizations Act, intentional or negligent infliction of
emotional or mental distress, tortious interference with corporate governments
or prospective business advantage, tortious interference with contractual
relations, breach of contract, deceptive trade practices, libel, slander,
conspiracy, the charging, contracting for, taking, reserving, collecting or
receiving of interest in excess of the highest lawful rate applicable to the
Loan Documents (i.e., usury), any violations of federal or state law, any
violations of federal or state banking rules, laws or regulations, including,
but not limited to, any violations of Regulation B, Equal Credit Opportunity,
bank tying act claims, any violation of the Texas Free Enterprise Antitrust Act
or any violation of federal antitrust acts.

     Section 7. SEVERABILITY. In the event any one or more provisions contained
in the Credit Agreement, this Amendment, or any of the Loan Documents should be
held to be invalid, illegal or unenforceable in any respect, the validity,
enforceability and legality of the remaining provisions contained herein and
therein shall not be affected in any way or impaired thereby and shall be
enforceable in accordance with their respective terms.

     Section 8. EXPENSES. The Borrowers agree to pay all out-of-pocket costs and
expenses (including reasonable fees and expenses of legal counsel) of Lender in
connection with the preparation, operation, administration and enforcement of
this Amendment.

     Section 9. ACKNOWLEDGMENT. Except as amended hereby, the Borrowers ratify
and confirm that the Loan Documents are and remain in full force and effect in
accordance with their respective terms and that all Collateral is unimpaired by
this Amendment and secures the payment and performance of all indebtedness and
obligations of the Borrowers under the Credit Note, the Credit Agreement, and
all other Loan Documents, as modified hereby. Each of the undersigned officers
of the Borrowers represent and warrant that his or her execution and delivery of
this Amendment has been duly authorized, and that the resolutions and affidavits
previously delivered to Lender, in connection with the execution and delivery of
the Credit Agreement and the First


                                                                               3

<PAGE>   4

Amendment thereto, are and remain in full force and effect and have not been
altered, amended or repealed in anywise.

     Section 10. NO WAIVER. The Borrowers agree that no Event of Default and no
Default has been waived or remedied by the execution of this Amendment by
Lender, and any such Default or Event of Default heretofore arising and
currently continuing shall continue after the execution and delivery hereof.

     Section 11. GOVERNING LAW. This Amendment shall be governed by and
construed in accordance with the laws of the State of Texas and, to the extent
applicable, by federal law.

     Section 12. COUNTERPARTS. This Amendment may be executed in any number of
counterparts and all of such counterparts taken together shall be deemed to
constitute one and the same instrument.

     SECTION 13. NO ORAL AGREEMENTS. THIS WRITTEN AMENDMENT, THE CREDIT
AGREEMENT, THE CREDIT NOTE, AND THE OTHER LOAN DOCUMENTS, ALL AS MODIFIED
HEREBY, REPRESENT THE FINAL AGREEMENT BETWEEN THE PARTIES AND MAY NOT BE
CONTRADICTED BY EVIDENCE OF PRIOR, CONTEMPORANEOUS OR ORAL AGREEMENTS OF THE
PARTIES.

           THERE ARE NO UNWRITTEN ORAL AGREEMENTS BETWEEN THE PARTIES.

                         [SIGNATURES ON FOLLOWING PAGE]


                                                                               4
<PAGE>   5



         EXECUTED and effective as of the dates first written above.


BORROWERS:                               HANOVER SPC-A, INC., a Delaware
                                         corporation


HANOVER CAPITAL MORTGAGE                 By: ___________________________________
HOLDINGS, INC.,                          Name:__________________________________
a Maryland corporation                   Title:_________________________________

By:________________________________      HANOVER CAPITAL SPC, INC., a Delaware
Name:______________________________      corporation
Title:_____________________________

                                         By:____________________________________
                                         Name:__________________________________
                                         Title:_________________________________
HANOVER CAPITAL PARTNERS, LTD.,
a New York corporation                   HANOVER REPO CORP., a Delaware
                                         corporation
By:________________________________
Name:______________________________      By:____________________________________
Title:_____________________________      Name:__________________________________
                                         Title:_________________________________

HANOVER QRS-1 98-B, INC., a Delaware
corporation

By:________________________________
Name:______________________________
Title:_____________________________      LENDER:

HANOVER QRS-2 98-B, INC., a Delaware     BANK UNITED,
corporation                              a federal savings bank

By:________________________________      By:____________________________________
Name:______________________________      Name:__________________________________
Title:_____________________________      Title:_________________________________


                                                                               5