<PAGE>   1
                                                                   EXHIBIT 10.80



                              FIFTH AMENDMENT TO
                 FIRST AMENDED AND RESTATED CREDIT AGREEMENT


      THIS FIFTH AMENDMENT TO FIRST AMENDED AND RESTATED CREDIT
AGREEMENT (the "Amendment"), dated as of June 26, 1998, is entered into between
the Lenders party hereto, BANKERS TRUST COMPANY, a New York banking corporation,
as agent for the Lenders (the "Agent"), DORAL FINANCIAL CORPORATION (formerly
known as First Financial Caribbean Corporation), a corporation organized under
the laws of the Commonwealth of Puerto Rico ("DFC"), and DORAL MORTGAGE
CORPORATION, a corporation organized under the laws of the Commonwealth of
Puerto Rico and a wholly-owned subsidiary of DFC ("DMC", and together with DFC,
each a "Borrower" and collectively, the "Borrowers"), with reference to the
First Amended and Restated Credit Agreement, dated as of September 25, 1996,
between the Borrowers, the Agent and the lenders party thereto (as amended by
the First Amendment to First Amended and Restated Credit Agreement and Certain
Other Loan Documents dated as of January 8, 1997, the Second Amendment to First
Amended and Restated Credit Agreement dated as of March 28, 1997, the Third
Amendment to First Amended and Restated Credit Agreement dated as of June 27,
1997, and the Fourth Amendment to First Amended and Restated Credit Agreement
dated as of November 5, 1997, and as further amended, supplemented or otherwise
modified from time to time, the "Credit Agreement"). All capitalized terms used
but not otherwise defined herein shall have the meanings given such terms in the
Credit Agreement.

      The Lenders, the Agent and the Borrowers wish to amend the Credit
Agreement as set forth herein.

      ACCORDINGLY, the parties hereto agree as follows:

      Section 1.   Amendments to Credit Agreement.

      (a) The Facility 1 Commitment and Facility 2 Commitment of each Lender
shall be amended as set forth opposite such Lender's name on the signature pages
hereto, as each such Commitment may be reduced from time to time pursuant to
Section 8.6(c) of the Credit Agreement.

      (b) The following new definitions shall be added to Section 1.1 of the
Credit Agreement in appropriate alphabetical order:

            "RECOURSE SERVICING PORTFOLIO REPORT" shall mean a report prepared
            by the Borrowers in substantially the form set forth in Exhibit T-2,
            including the following information: the recourse Mortgage Loan or
            Mortgage-Backed Security investor; the type of recourse (i.e.
            limited or full); delinquency and foreclosure rates on the
            Borrowers' recourse servicing portfolio; the amount of risk weighted
            capital required to support DFC's consolidated recourse exposure;
            and consolidated total capital to risk weighted assets in both
            amount and ratio terms.




<PAGE>   2





            "SERVICING COMMITMENTS" shall mean the "Commitments" as such term is
            defined in the Servicing Credit Agreement.

            "SERVICING EVENT OF DEFAULT" shall mean an "Event of Default" as
            such term is defined in the Servicing Credit Agreement.

            "SERVICING OBLIGATIONS" shall mean the "Obligations" as such term is
            defined in the Servicing Credit Agreement.

            "YEAR 2000 ISSUE" shall mean the failure of computer software,
            hardware and firmware systems and equipment containing embedded
            computer chips to properly receive, transmit, process, manipulate,
            store, retrieve, re-transmit or in any other way utilize data and
            information due to the occurrence of the year 2000 or the inclusion
            of dates on or after January 1, 2000.


      (c) The following definitions in Section 1.1 of the Credit Agreement shall
be amended to read as follows:

            "COLLATERAL VALUE OF THE PLEDGED SERVICING PORTFOLIO" shall mean, at
            the time of determination thereof, an amount equal to the lesser of
            (i) 70% of the fair market value of the servicing rights relating to
            the mortgage loans included in the Pledged Servicing Portfolio and
            (ii) 1.10% of the unpaid principal balance of the mortgage loans
            included in the Pledged Servicing Portfolio (in each case as
            reflected on the most recent Pledged Servicing Valuation Report
            delivered to the Agent).

            "EURODOLLAR INTEREST PERIOD" shall mean, with respect to any
            Eurodollar Loan, the period commencing on the Borrowing Date or a
            Conversion/Continuation Date for such Eurodollar Loan, as the case
            may be, and ending one, two or three months thereafter as the
            Borrowers may elect in the applicable Notice of Borrowing or Notice
            of Conversion/Continuation; provided that (a) any Eurodollar
            Interest Period that would otherwise end on a day that is not a
            Eurodollar Business Day shall be extended to the next succeeding
            Eurodollar Business Day, unless such Eurodollar Business Day falls
            in another calendar month, in which case such Eurodollar Interest
            Period shall end on the next preceding Eurodollar Business Day; (b)
            any Eurodollar Interest Period that begins on the last Eurodollar
            Business Day of a calendar month or any Eurodollar Interest Period
            that begins on a day for which there is no numerically corresponding
            day in the calendar month at the end of such Eurodollar Interest
            Period shall end on the last Eurodollar Business Day of such
            calendar month at the end of such Eurodollar Interest Period; and
            (c) no Eurodollar Interest Period shall end after the Maturity Date.





                                       2
<PAGE>   3




            "MATURITY DATE" shall mean June 25, 1999; provided that upon the
            written request of the Borrowers to the Agent, the Lenders may elect
            to extend the Maturity Date on terms as they may deem appropriate in
            their sole discretion.

            "OBLIGATIONS" shall mean, collectively, (i) the unpaid principal of
            and interest on the Loans and any Notes and all other obligations
            and liabilities of each of the Borrowers to the Agent, the
            Collateral Agents and the other Secured Parties (including, without
            limitation, interest accruing at the then applicable rate provided
            in this Agreement after the maturity of the Loans and interest
            accruing at the then applicable rate provided in this Agreement
            after the filing of any petition in bankruptcy, or the commencement
            of any insolvency, reorganization or like proceeding, relating to
            either Borrower, whether or not a claim for post-filing or
            post-petition interest is allowed in such proceeding), whether
            direct or indirect, absolute or contingent, due or to become due,
            voluntary or involuntary, whether or not jointly owed with others,
            liquidated or unliquidated, or now existing or hereafter incurred,
            and whether or not from time to time decreased or extinguished and
            later increased, created or incurred, which may arise under, out of,
            or in connection with, this Agreement, the Loans, any Notes, the
            Security Agreements, the other Loan Documents or any other document
            made, delivered or given in connection therewith, in each case
            whether on account of principal, interest, reimbursement
            obligations, fees, indemnities, costs, expenses or otherwise
            (including, without limitation, all fees and disbursements of
            counsel to any of the Agent, the Collateral Agents or any of the
            other Secured Parties that are required to be paid by the Borrowers
            pursuant to the terms of this Agreement, the Security Agreements or
            any other Loan Document), and (ii) the Servicing Obligations;
            provided that any reference in the Loan Documents to acceleration of
            the Obligations shall be deemed to refer only to the acceleration of
            the obligations referred to in clause (i) above.

            "PLEDGED SERVICING PORTFOLIO REPORT" shall mean a report prepared by
            the Borrowers in the format prescribed by the Mortgage Bankers'
            Association of America listing the FNMA/GNMA /FHLMC statistics and
            the other information shown on Exhibit T-1.

            "SERVICING CREDIT AGREEMENT" shall mean the First Amended and
            Restated Credit Agreement, dated as of June 26, 1998, between the
            Borrowers, Bankers Trust Company, as Agent thereunder, and the
            lenders party thereto from time to time, as amended, supplemented or
            otherwise modified from time to time.

            "SERVICING SECURITY AGREEMENT" shall mean the First Amended and
            Restated Security Agreement dated as of June 26, 1998, between the
            Servicing Collateral Agent and the Borrowers, as amended,
            supplemented or otherwise modified from time to time.


                                       3
<PAGE>   4



      (d) The definition of "BOOK NET WORTH" set forth in Section 1.1 of the
Credit Agreement and Sections 3.1(b)(iv), 4.4(a) and 4.4(d) of the Credit
Agreement shall be amended by replacing "December 31, 1995" with "December 31,
1997".

      (e) The definition of "ELIGIBLE MORTGAGE LOAN" set forth in Section 1.1 of
the Credit Agreement shall be amended by (i) deleting the "and" at the end of
clause (o) thereof, (ii) changing "." at the end of clause (p) thereof to ";
and", and (iii) adding the following new clause (q) immediately following clause
(p) thereof:

            "(q) such Mortgage Loan is not a "B or C mortgage loan" (as such
            term is customarily used in the mortgage banking industry), provided
            that any Mortgage Loan that is rated "B" or higher and that
            qualifies for insurance by a nationally recognized mortgage
            insurance company shall not be excluded as an Eligible Mortgage Loan
            as a result of this clause (q) unless the collateral value of such
            Mortgage Loan, when added to the collateral value of all other
            Mortgage Loans then included as Eligible Mortgage Loans under this
            clause (q), exceeds fifteen percent (15%) of the aggregate Facility
            1 Commitments."

      (f) Section 2.1(d) of the Credit Agreement shall be amended by replacing
"$3,000,000" in clause (ii) thereof with "$5,000,000".

      (g) Section 4.4(b) of the Credit Agreement shall be amended by replacing
(i) "June 30, 1996" with "April 30, 1998", and (ii) "six months" with "four
months".

      (h) The following new Section 4.15 shall be added to the Credit Agreement
immediately following Section 4.14 thereof:

            "SECTION 4.15  YEAR 2000 ISSUE

            Each Borrower and its Subsidiaries have reviewed the effect of the
Year 2000 Issue on the computer software, hardware and firmware systems and
equipment containing embedded microchips owned or operated by or for such
Borrower and its Subsidiaries or used or relied upon in the conduct of their
business (including systems and equipment supplied by others or with which such
computer systems of such Borrower and its Subsidiaries interface). The costs to
such Borrower and its Subsidiaries of any reprogramming required as a result of
the Year 2000 Issue to permit the proper functioning of such systems and
equipment and the proper processing of data, and the testing of such
reprogramming, and of the reasonably foreseeable consequences of the Year 2000
Issue to such Borrower or any of its Subsidiaries (including reprogramming
errors and the failure of systems or equipment supplied by others) are not
reasonably expected to result in a Potential Default or Event of Default or to
have a material adverse effect on the business, assets, operations, prospects or
condition (financial or otherwise) of such Borrower or any of its Subsidiaries."


                                       4
<PAGE>   5

      (i) Section 5.1(a)(x)(C) of the Credit Agreement shall be amended to read
as follows:

            "(C) As soon as available and in any event no later than thirty (30)
            days after the end of each six-month period ended June 30 and
            December 31 of each year, commencing with the six-month period
            ending June 30, 1998, a Pledged Servicing Portfolio Report dated as
            of the last day of such six-month period;"

      (j) Section 5.1(a)(x) of the Credit Agreement shall be further amended by
(i) deleting the word "and" from the end of clause (E) thereof, (ii) changing
"." at the end of clause (F) thereof to "; and", and (iii) adding the following
new clause (G) immediately following clause (F) thereof:

            "(G) As soon as available and in any event no later than thirty (30)
            days after the end of each quarterly period ended March 31, June 30,
            September 30 and December 31 of each year, commencing with the
            quarterly period ending June 30, 1998, a Recourse Servicing
            Portfolio Report dated as of the last day of such quarterly period."


      (k) Section 5.1(i) of the Credit Agreement shall be amended to read as
follows:

            "(i) Year 2000 Issue. Take, and shall cause each of its Subsidiaries
            to take, all necessary action to complete in all materials respects
            by June 30, 1999, the reprogramming of computer software, hardware
            and firmware systems and equipment containing embedded microchips
            owned or operated by or for such Borrower and its Subsidiaries or
            used or relied upon in the conduct of their business (including
            systems and equipment supplied by others or with which such systems
            of such Borrower or any of its Subsidiaries interface) required as a
            result of the Year 2000 Issue to permit the proper functioning of
            such computer systems and other equipment and the testing of such
            systems and equipment, as so reprogrammed. At the request of the
            Agent, such Borrower shall provide, and shall cause each of its
            Subsidiaries to provide, to the Agent reasonable assurance of its
            compliance with the preceding sentence."

      (l) Section 5.2(o) of the Credit Agreement shall be amended to read as
follows:

            "(o) Negative Pledges. Unless (i) the Facility 2 Commitments and the
            Servicing Commitments have been terminated by the Borrowers, (ii)
            all Facility 2 Loans and related Obligations and Servicing Loans and
            Servicing Obligations have been repaid or paid, respectively, by the
            Borrowers, (iii) no Potential Default or Event of Default has
            occurred and is continuing, and (iv) the Borrowers and the Lenders
            have agreed 


                                       5
<PAGE>   6

            upon the terms and conditions upon which this Agreement and the
            Facility 1 Commitments shall remain in effect (as contemplated in
            Section 2.7(b)), agree with any Person or Persons (other than with
            the Lenders pursuant to this Agreement and the Servicing Lenders
            pursuant to the Servicing Credit Agreement) not to create, incur,
            assume or suffer to exist, or permit any Subsidiary to create,
            incur, assume or suffer to exist, any Lien upon the Servicing
            Portfolio or any rights relating thereto, including, without
            limitation, any rights to receive payments in connection with the
            Servicing Portfolio (including, without limitation, any
            reimbursement of principal and interest advances, taxes and
            insurance advances and any other servicing advances)."

      (m) The following new Section 5.2(p) shall be added to the Credit
Agreement immediately following Section 5.2(o) thereof:

            "(p) Recourse Obligations. In the case of DFC, without the prior
            written consent of the Agent, acting at the direction of the
            Required Lenders, enter into any transaction whereby (i) DFC sells
            Mortgage-Backed Securities or Mortgage Loans and remains obligated
            to repurchase (whether conditionally or unconditionally) such
            Mortgage-Backed Securities or Mortgage Loans or has retained other
            recourse obligations with respect thereto, or (ii) DFC assumes or
            otherwise incurs recourse obligations with respect to
            Mortgage-Backed Securities or Mortgage Loans included in its
            Servicing Portfolio, if, in either case, in the reasonable judgment
            of the management of DFC, the aggregate obligations of DFC in
            connection with such transaction and all previous transactions of
            the type described in clauses (i) and (ii) above for which DFC is
            then obligated could, if DFC is required to perform such
            obligations, result in a decrease in DFC's Book Net Worth by a
            Material Amount."

      (n) Section 5.3(a) of the Credit Agreement shall be amended to read as
follows:

                  "(a) Total Liabilities. Permit DFC on a consolidated basis
            (excluding any Subsidiaries that are not primarily engaged primarily
            in the business of mortgage banking as reasonably determined by the
            Agent) to incur Total Liabilities in excess of the sum of (i) one
            hundred percent (100%) of "Cash" or "Cash equivalents";
            (ii) ninety-five percent (95%) of the sum of (A) "Mortgage Loans
            held for sale", (B) "Mortgage-backed securities held for trading"
            (but excluding "interest only securities" included therein), and (C)
            "U.S. Treasury & Other AAA Rated Investments that are held to
            maturity or available for sale"; (iii) ninety percent (90%) of
            "Accrued interest receivable"; (iv) eighty percent (80%) of the sum
            of (A) "Mortgage-backed securities held to maturity", (B) "Accounts
            receivable and mortgage servicing advances", and (C)
            "Mortgage-backed securities available for sale"; (v) eighty percent
            (80%) of "prepaid and other assets" (excluding investment in
            Subsidiaries); (vi) fifty percent (50%) of the sum of (A) "Property,
            leasehold 


                                       6
<PAGE>   7

            improvements and equipment" and (B) "Real estate held for sale";
            (vii) sixty-five percent (65%) of "interest only securities"; and
            (viii) one percent (1.0%) of the principal amount of Mortgage Loans
            owned by Persons not affiliated with DFC or DMC or any of their
            Affiliates (unless covered by a Permitted Affiliate Servicing
            Agreement) for which DFC or DMC owns the direct servicing rights.
            All quoted terms used in the preceding sentence shall have the same
            meanings, and shall continue to be calculated and classified in the
            same manner, as the terms used in the balance sheet of DFC and its
            consolidated Subsidiaries referred to in Section 4.4(a)."

      (o) Section 5.3(b) of the Credit Agreement shall be amended to read as
follows:

                  "(b) Adjusted Tangible Net Worth. Permit Adjusted Tangible Net
            Worth at anytime to be less than $108,000,000."

      (p) Section 5.3(c) of the Credit Agreement shall be amended to read as
follows:

                  "(c) Book Net Worth. Permit Book Net Worth at any time to be
            less than $126,000,000."

      (q) Section 5.3(e) of the Credit Agreement shall be amended to read as
follows:

            "(e) Servicing Portfolio. Permit DFC on a consolidated basis
            (excluding any Subsidiaries that are not primarily engaged in the
            business of mortgage banking as reasonably determined by the Agent)
            to maintain a Servicing Portfolio of Mortgage Loans with an
            aggregate outstanding principal balance of less than $3,500,000,000
            or such lesser amount as shall be agreed by the Lenders in their
            sole discretion."

      (r) Section 6.1 of the Credit Agreement shall be amended by adding the
following new subparagraph (k) immediate following subparagraph (j) thereof and
relettering subparagraphs "(k)" and "(l)" thereof as subparagraphs "(l)" and
"(m)", respectively:

            "(k) Servicing Event of Default. A Servicing Event of Default shall
occur;"

      (s) The first sentence of Section 8.6(c) of the Credit Agreement shall be
amended to read as follows:

            "Subject to the limitations set forth below, no Lender may assign
            its rights or delegate its obligations under this Agreement and the
            other Loan Documents to any other financial institution without the
            prior consent of the Agent and the Borrowers (such consent not to be
            unreasonably withheld); provided that (i) no Lender may make an
            assignment hereunder to a Person (other than an Affiliate of such
            Lender or an existing Lender) that is not a financial institution,
            (ii) each assignment shall 


                                       7
<PAGE>   8

            include a corresponding portion of such Lender's Commitments, Loans,
            Servicing Commitments and Servicing Loans, it being the intent of
            the Lenders that each Lender and each subsequent Lender added
            hereunder by way of assignment or otherwise shall have and maintain
            the same pro rata interest in all Commitments, Loans, Servicing
            Commitments and Servicing Loans, and (iii) an assignment fee in the
            amount of $3,500 for each assignment hereunder shall be payable to
            the Agent by the applicable assignee."

      (t) Exhibit K to the Credit Agreement shall be amended by amending clause
(b)(i) thereof to read as follows:

            "(i)  No prepayment or amortization of the principal amount of any 
                  subordinate debt (the "SUBORDINATE DEBT") shall be permitted;"

      (u) Exhibit N to the Credit Agreement shall be amended by adding Mario S.
Levis as an Authorized Officer of Doral Mortgage Corporation.

      (v) Exhibit T to the Credit Agreement shall be re-labelled "Exhibit T-1"
and Exhibit T-2 shall be added to the Credit Agreement in the form of Exhibit
T-2 hereto.

      Section 2. Acknowledgement of Obligations under Loan Documents. The
Borrowers hereby acknowledge and agree that any reference to the "Obligations"
in the Loan Documents, including, without limitation, each of the Security
Agreements, shall refer to the "Obligations" as defined in Section 1(c) of this
Amendment.

      Section 3. Representations and Warranties. The Borrowers represent and
warrant that, on and as of the date hereof, all of the representations and
warranties made by them in the Credit Agreement and the other Loan Documents are
true and correct as if made on and as of the date hereof and no Potential
Default or Event of Default has occurred and is continuing.

      Section 4. Effectiveness. This Amendment shall become effective as of the
date hereof upon delivery to the Agent of (i) counterparts of this Amendment and
of the Servicing Credit Agreement, duly executed and delivered by the parties
hereto and thereto, together with all related instruments, documents and
agreements required to be delivered hereunder and thereunder, and (ii)
additional Security Agreements in substantially the form of Exhibits C, D and E
to the Credit Agreement, duly executed and delivered by the parties thereto,
provided that the execution and delivery of such Security Agreements shall not
affect any of the Security Agreements in effect as of the date of this
Amendment, which Security Agreements shall remain in full force and effect.

      Section 5. Intercreditor Agreement. Each of the Agent, the Servicing
Collateral Agent and the Lenders agrees that the Intercreditor Agreement dated
as of November 5, 1997, between the Agent, the Servicing Collateral Agent, the
Lenders and each of the Servicing Secured Parties (as 


                                       8
<PAGE>   9

defined therein), as amended, supplemented or otherwise modified to the date
hereof, shall be terminated as of the effective date of this Amendment without
any further action by any party hereto or thereto.

      Section 6. Counterparts. This Amendment may be executed in any number of
counterparts, all of which taken together shall constitute one agreement, and
any party hereto may execute this Amendment by signing any such counterpart.

      Section 7.  GOVERNING LAW.  THIS AMENDMENT SHALL BE GOVERNED BY
AND CONSTRUED IN ACCORDANCE WITH THE LAWS OF THE STATE OF NEW
YORK.

      Section 8. Miscellaneous. Except as expressly amended hereby, the Credit
Agreement and the other Loan Documents shall remain in full force and effect.
Nothing contained herein shall operate as a waiver of any right, power or remedy
of the Agent or the Lenders under the Credit Agreement or any other Loan
Document, nor constitute a waiver of any provision of the Credit Agreement or
any other Loan Document.








                                       9
<PAGE>   10





      IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
executed as of the day and year first above written.


                              DORAL FINANCIAL CORPORATION,
                              as a Borrower

                              By:   /s/ MARIO S. LEVIS
                                    ----------------------------------------- 
                              Name:  Mario S. Levis
                              Title: Executive Vice President and Treasurer


                              DORAL MORTGAGE CORPORATION,
                              as a Borrower

                              By:   /s/ MARIO S. LEVIS
                                    -----------------------------------------
                              Name:  Mario S. Levis
                              Title: Executive Vice President and Treasurer


Facility 1 Commitment:        BANKERS TRUST COMPANY,
$25,000,000                   as Agent and as a Lender

Facility 2 Commitment:        By:   /s/ KEVIN M. MCCANN
$2,000,000                          -----------------------------------------   
                              Name: Kevin M. McCann
                              Title:Principal


Facility 1 Commitment:        FIRST UNION NATIONAL BANK,
$25,000,000                   as a Lender

Facility 2 Commitment:
$2,000,000                    By:   /s/ R. STEVEN HALL
                                    -----------------------------------------
                              Name: R. STEVEN HALL
                                    -----------------------------------------
                              Title:Vice President
                                    -----------------------------------------






                                       10
<PAGE>   11


Facility 1 Commitment:        BANKBOSTON, N.A. (formerly known as
$25,000,000                   The Bank of Boston),
                              as a Lender
Facility 2 Commitment:
$2,000,000                    By:   /s/ PAUL CHMIELINSKI
                                    ------------------------------------- 
                              Name: PAUL CHMIELINSKI
                                    -------------------------------------
                              Title:Vice President
                                    -------------------------------------



Facility 1 Commitment:        THE BANK OF NEW YORK,
$25,000,000                   as a Lender

Facility 2 Commitment:        By:   /s/ ROBERT A. TWEED
                                    -------------------------------------
$2,000,000                    Name: ROBERT A. TWEED
                                    -------------------------------------
                              Title:Vice President
                                    -------------------------------------



Facility 1 Commitment:        NATIONAL CITY BANK OF KENTUCKY,
$25,000,000                   as a Lender

Facility 2 Commitment:        By:   /s/ ROBERT J. OGBURN
                                    -------------------------------------
$2,000,000                    Name: ROBERT J. OGBURN
                                    -------------------------------------
                              Title:Vice President
                                    -------------------------------------





                                       11
<PAGE>   12




                                                                   EXHIBIT L-2

                 FORM OF RECOURSE SERVICING PORTFOLIO REPORT






                                       12