<PAGE>

                                                                    Exhibit 10.4

                ASSIGNMENT, ASSUMPTION AND RECOGNITION AGREEMENT

      THIS ASSIGNMENT, ASSUMPTION AND RECOGNITION AGREEMENT (this "Assignment"),
dated as of June 1, 2004, is entered into among J.P. Morgan Acceptance
Corporation I, a Delaware corporation (the "Depositor"), Wachovia Bank, National
Association, as trustee (the "Trustee") of J.P. Morgan Mortgage Trust 2004-A3
(the "Trust"), Sunset Financial Resources, Inc., a Maryland corporation
("Sunset"), Cendant Mortgage Corporation ("Cendant") and Bishop's Gate
Residential Mortgage Trust ("Bishop's Gate" and, together with Cendant, the
"Sellers"), with Cendant Mortgage Corporation, as the servicer (in such
capacity, the "Servicer").

                                    RECITALS

      WHEREAS the Sellers and the Servicer and Morgan Stanley Mortgage Capital
Inc., as purchaser ("Morgan Stanley"), entered into a certain Mortgage Loan Flow
Purchase, Sale & Servicing Agreement, dated as of November 13, 2003 (the
"Agreement"), pursuant to which Morgan Stanley acquired certain mortgage loans
(the "Mortgage Loans") pursuant to the terms of the Agreement and the Servicer
agreed to service such Mortgage Loans;

      WHEREAS, pursuant to an Assignment, Assumption and Recognition Agreement
dated March 30, 2004 relating to the Agreement, Sunset acquired from Morgan
Stanley all of Morgan Stanley's right, title and interest in certain of the
Mortgage Loans (the "Sunset Mortgage Loans") and the Servicer agreed to service
such Sunset Mortgage Loans pursuant to the Agreement;

      WHEREAS the Depositor has agreed, on the terms and conditions contained
herein, to purchase from Sunset certain of the Sunset Mortgage Loans (the
"Specified Mortgage Loans") which are subject to the provisions of the Agreement
and are listed on the mortgage loan schedule attached as Exhibit I hereto (the
"Specified Mortgage Loan Schedule"); and

      WHEREAS the Trustee, on behalf of the Trust, has agreed, on the terms and
conditions contained herein, to purchase from the Depositor the Specified
Mortgage Loans;

      NOW, THEREFORE, in consideration of the mutual promises contained herein
and other good and valuable consideration (the receipt and sufficiency of which
are hereby acknowledged), the parties agree as follows:

      1. Assignment and Assumption

      (a) On and as of the date hereof, Sunset hereby sells, assigns and
transfers to the Depositor all of its right, title and interest in the Specified
Mortgage Loans and all rights related thereto as provided under the Agreement to
the extent relating to the Specified Mortgage Loans (without recourse or
warranty except as otherwise specifically provided herein), the Depositor hereby
accepts such assignment from Sunset (the "First Assignment and Assumption"), and
the Sellers hereby acknowledge the First Assignment and Assumption.

          Sunset specifically reserves and does not assign to the Depositor
hereunder any and all right, title and interest in, to and under and all
obligations of Sunset with respect to any Sunset Mortgage Loans subject to the
Agreement which are not the Specified Mortgage Loans.

      (b) On and as of the date hereof, immediately after giving effect to the
First Assignment and Assumption, the Depositor hereby sells, assigns and
transfers to the Trustee, on behalf of the Trust, all of

<PAGE>

its right, title and interest in the Specified Mortgage Loans and all rights
related thereto as provided under the Agreement to the extent relating to the
Specified Mortgage Loans, and the Trustee, on behalf of the Trust, hereby
accepts such assignment from the Depositor (the "Second Assignment and
Assumption"), and the Sellers hereby acknowledge the Second Assignment and
Assumption.

      (c) On and as of the date hereof, Sunset represents and warrants to the
Depositor and the Trustee that Sunset has not taken any action that would serve
to impair or encumber the respective ownership interests of the Depositor and
the Trustee in the Specified Mortgage Loans since the date of Sunsets'
acquisition of the Specified Mortgage Loans.

      2. Recognition of Trustee

      (a) From and after the date hereof, both Sunset and the Sellers shall note
the transfer of the Specified Mortgage Loans to the Trustee, in their respective
books and records and shall recognize the Trustee, on behalf of the Trust, as of
the date hereof, as the owner of the Specified Mortgage Loans, and Servicer
shall service the Specified Mortgage Loans for the benefit of the Trust pursuant
to the Agreement, the terms of which are incorporated herein by reference. It is
the intention of the Sellers, the Servicer, the Depositor, the Trustee and
Sunset that this Assignment shall be binding upon and inure to the benefit of
the Depositor, the Trustee and Sunset and their respective successors and
assigns.

      (b) Without in any way limiting the foregoing, the parties confirm that
this Assignment includes the rights relating to amendments and waivers under the
Agreement. Accordingly, the right of Sunset to consent to any amendment of the
Agreement and its rights concerning waivers as set forth in Section 12.07 of the
Agreement shall be exercisable, to the extent any such amendment or waiver
affects the Specified Mortgage Loans or any of the rights under the Agreement
with respect thereto (including, without limitation, the servicing of the
Specified Mortgage Loans), by the Trustee as assignee of Sunset.

      3. Representations and Warranties

      (a) The Depositor represents and warrants that it is a sophisticated
investor able to evaluate the risks and merits of the transactions contemplated
hereby, and that it has not relied in connection therewith upon any statements
or representations of the Sellers or Sunset other than those contained in the
Agreement or this Assignment.

      (b) Each of the parties hereto represents and warrants that it is duly and
legally authorized to enter into this Assignment.

      (c) Each of the parties hereto represents and warrants that this
Assignment has been duly authorized, executed and delivered by it and (assuming
due authorization, execution and delivery thereof by each of the other parties
hereto) constitutes its legal, valid and binding obligation, enforceable against
it in accordance with its terms, except as such enforcement may be limited by
bankruptcy, insolvency, reorganization or other similar laws affecting the
enforcement of creditors' rights generally and by general equitable principles
(regardless of whether such enforcement is considered in a proceeding in equity
or at law).

      (d) Each Seller hereby restates, as of Closing Date (as defined in the
Pooling and Servicing Agreement referred to below) (or, with respect to the
representations and warranties contained in Sections 3.03(4), (20), (21), (25),
(31) and (58) of the Agreement, as of the applicable Funding Date (as defined in
the Agreement)), the representations and warranties set forth in Article III of
the Agreement, with respect to each of the Specified Mortgage Loans that were
sold by it under the Agreement, to and for the benefit of the Depositor, the
Trustee and the Trust, and by this reference incorporates such representations
and

                                       2

<PAGE>

warranties herein, as of such Closing Date. In addition, each Seller hereby
represents and warrants that each Specified Mortgage Loan sold by it under the
Agreement that is a Cooperative Loan is secured by stock in a "cooperative
housing corporation" within the meaning of Section 216(b) of the Code.

      4. The Servicer hereby acknowledges that Wells Fargo Bank, N.A. has been
appointed as the master servicer (the "Master Servicer") of the Specified
Mortgage Loans pursuant to the pooling and servicing agreement dated as of the
date hereof among the Depositor, the Master Servicer, the Securities
Administrator and the Trustee (the "Pooling and Servicing Agreement") for
Mortgage Pass-Through Certificates, Series 2004-A3 and, therefore, has the right
to enforce all obligations of the Servicer under the Agreement with respect to
the Specified Mortgage Loans. Such rights will include, without limitation, the
right to terminate the Servicer under the Agreement with respect to the
Specified Mortgage Loans upon the occurrence of an event of default thereunder,
the right to receive all remittances required to be made by the Servicer under
the Agreement with respect to the Specified Mortgage Loans, the right to receive
all monthly reports and other data required to be delivered by the Servicer
under the Agreement with respect to the Specified Mortgage Loans, the right to
examine the books and records of the Servicer, indemnification rights and the
right to exercise certain rights of consent and approval relating to actions
taken by Sunset with respect to the Specified Mortgage Loans. The Servicer shall
make all distributions under the Agreement to the Master Servicer by wire
transfer of immediately available funds to:

                  Wells Fargo Bank, N.A.
                  ABA Number: 121-000-248
                  Account Name: SAS Clearing
                  Account number: 3970771416
                  For further credit to: J.P. Morgan Mortgage Trust 2004-A3,
                                         Distribution Account Number: 18172300

      The Servicer shall deliver all reports required to be delivered under the
Agreement to the Master Servicer at the following address:

                  Wells Fargo Bank, N.A.
                  9062 Old Annapolis Road
                  Columbia, Maryland 21045
                  Attention: Client Manager
                  Telecopier: (410) 715-2380

      5. Establishment of Collection Account

      The Servicer shall establish and maintain a separate Collection Account in
the name of the Trustee, in trust for J.P. Morgan Mortgage Trust 2004-A3, for
all funds collected and received on the Specified Mortgage Loans.

      6. Amendments to the Agreement

      The parties to the Agreement hereby agree to amend the Agreement as
follows:

      (a) The definition of "Qualified Substitute Mortgage Loan" is hereby
amended solely with respect to the Specified Mortgage Loans by deleting clause
(x) in its entirety and replacing it as follows:

            "(x) with respect to each ARM Loan, have the same Index as that of
            the Deleted Mortgage Loan,"

                                       3

<PAGE>

      (b) The definition of "Qualified Substitute Mortgage Loan" is hereby
amended solely with respect to the Specified Mortgage Loans by inserting the
following:

            "and (xiv) with respect to each ARM Loan, not permit conversion of
            the related Note Rate to a permanent fixed Note Rate."

      (c) With respect to the Specified Mortgage Loans, "Permitted Investments"
shall have the meaning of such term as defined in the Pooling and Servicing
Agreement.

      (d) Notwithstanding any provision in the Agreement to the contrary, the
parties to this Assignment hereby agree that the Servicer's aggregate
obligations under Section 6.03(3) of the Agreement to make payments on account
of Prepayment Interest Shortfall Amounts with respect to a Mortgage Pool (as
defined in the Pooling and Servicing Agreement) shall be limited to the total
amount of servicing compensation that would be payable to the Servicer with
respect to such Mortgage Pool if no Principal Payment was made during the
related Due Period.

      (e) Section 6.02 is hereby amended by deleting the first sentence of the
first paragraph and replacing it with the following sentence:

      "On or before the 5th Business Day of each month during the term hereof,
      the Servicer shall deliver to the Master Servicer monthly accounting
      reports in the form of Exhibits 6.02(a) through 6.02(g) and Exhibit 13
      attached hereto, in Microsoft Excel format as requested by the Purchaser,
      with respect to the most recently ended Due Period."

      (f) The reporting requirements attached to this Assignment as Exhibit II
is hereby incorporated into the Agreement as new Exhibit 13.

      (g) The references to "March 15" in Section 7.04 are hereby deleted and
replaced with "March 1".

      (h) Section 7.04 of the Agreement is hereby amended and restated in  its
entirety as follows:

      "Annual Statement as to Compliance. (a) For so long as the Mortgage Loans
      are being master serviced by a master servicer in a securitization
      transaction (the "Master Servicer"), by the later of March 1 or 30 days
      prior to the date on which the Form 10-K is required to be filed with the
      SEC of each year (or if not a Business Day, the immediately preceding
      Business Day), an officer of the Servicer shall execute and deliver an
      Officer's Certificate to the Master Servicer for the benefit of such
      Master Servicer and its officers, directors and affiliates, in the form of
      Exhibit 11 hereto.

      (b) The Servicer shall indemnify and hold harmless the Master Servicer and
      its officers, directors, agents and affiliates from and against any
      losses, damages, penalties, fines, forfeitures, reasonable legal fees and
      related costs, judgments and other costs and expenses arising out of or
      based upon a breach by the Servicer or any of its officers, directors,
      agents or affiliates of its obligations under this Section 7.04 or the
      negligence, bad faith or willful misconduct of the Servicer in connection
      therewith. If the indemnification provided for herein is unavailable or
      insufficient to hold harmless the Master Servicer, then the Servicer
      agrees that it shall contribute to the amount paid or payable by the
      Master Servicer as a result of the losses, claims, damages or liabilities
      of the Master Servicer in such proportion as is appropriate to reflect the
      relative fault of the Master Servicer on the one hand and the Servicer on
      the other in connection with a breach of the

                                       4

<PAGE>

      Servicer's obligations under this Section 7.04 or the Servicer's
      negligence, bad faith or willful misconduct in connection therewith."

      (i) The word "or" is deleted from the end of Section 10.01(8), the word
"or" is added at the end of Subsection 10.01(9) and the following paragraph is
hereby incorporated into the Agreement as new Subsection 10.01(10):

            "(10) failure by the Servicer to duly perform, within the required
      time period, its obligations under Sections 7.04 and 7.05 which failure
      continues unremedied for a period of thirty (30) days after the date on
      which written notice of such failure, requiring the same to be remedied,
      shall have been given to the Servicer by any party to this Agreement or by
      any master servicer responsible for master servicing the Mortgage Loans
      pursuant to a Pass-Through Transfer of such Mortgage Loans;"

      (j) The following paragraph is hereby incorporated into the Agreement as
new Section 12.14:

      "Third Party Beneficiary. For purposes of Section 7.04 only, any Master
Servicer shall be considered a third party beneficiary to Section 7.04 and shall
be entitled to enforce the provisions of Section 7.04."

      7. Indemnification by Master Servicer

      The Master Servicer shall indemnify and hold harmless the Servicer and its
affiliates, and in each case, its officers, directors and agents from and
against any losses, damages, penalties, fines, forfeitures, reasonable legal
fees and related costs, judgments and other costs and expenses arising out of or
based upon a breach by the Master Servicer or any of its officers, directors,
agents or affiliates of its obligations in connection with the preparation,
filing and certification of any Form 10-K pursuant to the Pooling and Servicing
Agreement or the negligence, bad faith or willful misconduct of the Master
Servicer in connection therewith. In addition, the Master Servicer shall
indemnify and hold harmless the Servicer and its affiliates, and in each case,
its officers, directors and agents from and against any losses, damages,
penalties, fines, forfeitures, reasonable legal fees and related costs,
judgments and other costs and expenses arising out of or based upon a breach by
any Servicer (as defined in the Pooling and Servicing Agreement), other than the
Servicer (as defined herein), of its obligations in connection with any back-up
certification (or any other back-up documents) to any certification of any Form
10-K required to be provided by the Master Servicer, but solely to the extent
the Master Servicer receives amounts from such Servicer in connection with any
indemnification provided by such Servicer (in each case as defined in the
Pooling and Servicing Agreement) to the Master Servicer.

      8. Continuing Effect

      Except as contemplated hereby, the Agreement shall remain in full force
and effect in accordance with its terms.

      9. Governing Law

      This Assignment and the rights and obligations hereunder shall be
governed by and construed in accordance with the internal laws of the State of
New York.

      10. Notices

                                       5

<PAGE>

      Any notices or other communications permitted or required under the
Agreement to be made to Sunset, the Depositor, the Trustee, Cendant, Bishop's
Gate and the Servicer shall be made in accordance with the terms of the
Agreement and shall be sent to the Depositor and Trustee as follows:

      In the case of Sunset:

            Sunset Financial Resources, Inc.
            10245 Centurion Parkway North
            Suite 305
            Jacksonville, Florida  32256
            Attention: Byron Boston
            Telephone: 904-425-4351
            Facsimile: 904-425-4755

      In the case of the Depositor:

            J.P. Morgan Acceptance Corporation I
            270 Park Avenue
            New York, New York 10017
            Attention: J.P. Morgan Mortgage Trust 2004-A3

      In the case of the Trustee:

            Wachovia Bank, National Association
            401 South Tryon Street
            Charlotte, North Carolina 28288-1179
            Attention: Structured Finance Trust Services,
            J.P. Morgan Mortgage Trust 2004-A3

      In the case of Cendant:

            Cendant Mortgage Corporation
            3000 Leadenhall Road
            Mt. Laurel, NJ 08054
            Attention: Peter A. Thomas, Vice President, Secondary Marketing

      In the case of Bishop's Gate:

            c/o Cendant Mortgage Corporation, as Administrator
            3000 Leadenhall Road
            Mt. Laurel, NJ 08054
            Attention: Peter A. Thomas, Vice President, Secondary Marketing

      In the case of the Servicer:

            Cendant Mortgage Corporation
            3000 Leadenhall Road
            Mt. Laurel, NJ 08054
            Attention: Peter A. Thomas, Vice President, Secondary Marketing

                                       6

<PAGE>

or to such other address as may hereafter be furnished by the Depositor and the
Trustee to the parties in accordance with the provisions of the Agreement.

      11. Ratification

      Except as modified and expressly amended by this Assignment, the Agreement
is in all respects ratified and confirmed, and all terms, provisions and
conditions thereof shall be and remain in full force and effect.

      12. Counterparts

      This Assignment may be executed in counterparts, each of which when so
executed shall be deemed to be an original and all of which when taken together
shall constitute one and the same instrument.

      13. Definitions

      Any capitalized term used but not defined in this Assignment has the same
meaning as in the Agreement.

                            [SIGNATURE PAGE FOLLOWS]

                                       7

<PAGE>

      IN WITNESS WHEREOF, the parties hereto have executed this Assignment the
day and year first above written.

                                        SUNSET FINANCIAL RESOURCES, INC.

                                        By: /s/ Thomas G. Manuel
                                            ------------------------------
                                            Name: Thomas G. Manuel
                                            Title: EVP

                                        J.P. MORGAN ACCEPTANCE CORPORATION I

                                        By: /s/ Jonathan Davis
                                            ----------------------------------
                                            Name: Jonathan Davis
                                            Title: Vice President

                                        WACHOVIA BANK, NATIONAL
                                        ASSOCIATION, as Trustee of J.P. Morgan
                                        Mortgage Trust 2004-A3

                                        By: /s/ Stephanie J. Aronovitch
                                            ----------------------------------
                                            Name: Stephanie J. Aronovitch
                                            Title: Assistant Vice President

                                        CENDANT MORTGAGE CORPORATION

                                        By: /s/ Crissy Judge
                                            ----------------------------------
                                            Name: Crissy Judge
                                            Title: Assistant Vice President

                                        BISHOP'S GATE RESIDENTIAL
                                        MORTGAGE TRUST

                                        By: /s/ Crissy Judge
                                            -----------------------------------
                                            Name: Crissy Judge
                                            Title: Assistant Vice President

<PAGE>

                                        CENDANT MORTGAGE CORPORATION,as Servicer

                                        By: /s/ Crissy Judge
                                            -----------------------------------
                                            Name: Crissy Judge
                                            Title: Assistant Vice President

Acknowledged and Agreed:
WELLS FARGO BANK, N.A., as Master Servicer

By: /s/ Ruth Fussell
    -------------------------------
    Name: Ruth Fussell
    Title: Vice President