<PAGE>
 
                                                                    Exhibit 10.4


                          RETIREMENT PAYMENT AGREEMENT

AGREEMENT ENTERED INTO AS OF THE 1ST DAY OF August, 1987 between Haywood Savings
                                 ---        ------------         ---------------
and Loan Association, a domestic Corporation having its principal office in
--------------------                                                       
Waynesville, North Carolina (hereinafter referred to as the Association) and
-----------  --------------                                                 
Larry R. Ammons of Waynesville, North Carolina (hereinafter referred to as the
---------------    -----------  --------------                                
Director).

WITNESSETH:

WHEREAS, the Director is rendering valuable service and it is the desire of the
Association to have the benefit of his continued loyalty and service and also to
assist him in providing for the contingencies of retirement and death; and,

WHEREAS, the Director hereby agrees to waive fees paid to him as a Director in
the amount of $350 per month for five years from the date of the execution of
              ----               ----                                        
this agreement;

NOW THEREFORE, in consideration of the premises contained herein, the parties
hereto mutually agree as follows:

     1.  Retirement Benefit:  Should the Director still be in the Directorship
         ------------------                                                   
of the Association upon attainment of his 65th birthday, the Association will
                                          ----                               
commence to pay him $2,870 per month for a continuous period of 120 months.  In
                    ------                                      ---            
the event that the Director should die after becoming entitled to receive said
monthly installments but before any or all of said installments have been paid,
the Association will pay or will continue to pay said installments to such
beneficiary or beneficiaries as the Director has directed by filing with the
Association a notice in writing.  In the event of the death of the last named
beneficiary before all the unpaid payments have been made, the balance of any
amount which remains unpaid at said death shall be commuted on the basis of 6
percent per annum compound interest ann shall be paid in a single sum to the
executor or administrator of the estate of the last named beneficiary to die.
In the absence of any such beneficiary designation, any amount remaining unpaid
at the Director's death shall be commuted on the basis of eight percent per
annum compound interest and shall be paid in a single sum to the executor or
administrator of the Director's estate.

     2.  Death Benefit:  Should the Director die while in the Directorship of
         -------------                                                       
the Association and prior to the attainment of his 65th birthday, the
                                                   ----              
Association (beginning at a date to be determined by the Association but within
six months from the date of such death) will commence to pay $2,870 per month
                                                             ------          
for a continuous period of 120 months to such beneficiary or beneficiaries as
                           ---                                               
the Director has directed by filing with the Association a notice in writing.
Irrespective of the above, however, if the Director dies as a result of suicide
within two years of the execution of this agreement, the death benefit shall not
exceed an amount equal to his waived Directors' fees plus interest at the rate
of 7  1/2 percent per annum compounded annually.  In the event of the death of
the last named beneficiary before all the unpaid payments have been made, the
balance of any amount which remains unpaid at said death shall be commuted on
the basis of 6 percent per annum compound 
<PAGE>
 
interest and shall be paid in a single sum to the executor or administrator of
the estate of the last named beneficiary to die. In the absence of any such
beneficiary designation, any amount remaining unpaid at the Director's death
shall be commuted on the basis of 6 percent per annum compound interest and
shall be paid in a single sum to the executor or administrator of the Director's
estate.

3.  Termination of Directorship:
    --------------------------- 

    A.  If the Director terminates his Directorship, for reasons other than
    death or the attainment of his 65th birthday, prior to two years from the
                                   ----                                      
    execution date of the Agreement, the Director's benefits shall be limited
    to his waived Director fees plus interest at the rate of 7 1/2 percent per
    annum compounded annually and shall be paid in a single sum as soon as
    practical following the termination of his Directorship.

    B.  If the Director terminates his Directorship, for reasons other than
    death or the attainment of his 65th birthday, at the end of two years or
                                   ----                                     
    more from the execution date of this Agreement, he or his beneficiary, as
    applicable, shall be entitled upon the attainment of his 65th birthday, or
                                                             ----             
    his prior death, to a percentage of the retirement benefits stated in
    Section 1 of this Agreement as determined by the following table:

<TABLE>
<CAPTION> 


    FULL NUMBER OF YEARS SERVED        PERCENTAGE OF RETIREMENT
     AS DIRECTOR FROM DATE OF         BENEFITS STATED IN SECTION
    EXECUTION OF THIS AGREEMENT      1 OF THIS AGREEMENT TO WHICH
 UNTIL TERMINATION OF DIRECTORSHIP     THE DIRECTOR IS ENTITLED
-----------------------------------  ----------------------------
<S>                                  <C>
                                                               
                                                               
                 2                              40%               
                                                                  
                 3                              60%               
                                                                  
                 4                              80%   
                                                   
                 5                             100% 
</TABLE>

4.   Forfeiture Provisions:
     --------------------- 

     A.  During the period the retirement benefit is payable to the Director
     under Section 1 of this Agreement, the Director shall not engage in
     business activities which are in competition with the Association without
     first obtaining the written consent of the Association.

     B.  During the period the retirement payment is payable to the Director
     under Section 1 of the Agreement, the Director shall be available to render
     consulting services to the Association upon request by an officer of the
     Association, but such requests shall not be made  more frequently than once
     each moth.  The Director shall not be considered to have breached this
     condition if he is unable to consult because of his mental or physical
     disability.
<PAGE>
 
     C.  Payment of the retirement benefit under this Agreement may be
     terminated by the Association, if the Director fails to comply with either
     of the conditions set forth in paragraph (A) and (B) of this Section 4.


5.   General Provisions:
     ------------------ 

     A.  Except as otherwise provided by the Agreement, it is agreed that
     neither the Director, nor his beneficiary shall have any right to commute,
     sell, assign, transfer or otherwise convey the right to receive any
     payments hereunder, which payments and the right hereto are expressly
     declared to be nonassignable and nontransferable.

     B.  The benefits payable under the Agreement shall be independent of, and
     in addition to, any other employment agreements that may exist from time to
     time between the parties hereto, concerning any other compensation payable
     by the Association to the Director whether as salary, bonus, or otherwise.
     This Agreement shall not be deemed to constitute a contract of employment
     between the parties hereto, nor shall any provision hereof restrict the
     right of the Association to discharge the Director or restrict the right of
     the Director to terminate his Directorship.

     C.  The rights of the Director under this Agreement and of any beneficiary
     of the Director shall be solely those of an unsecured creditor of the
     Association.  Any asset acquired by the Association in connection with the
     liabilities assumed by it hereunder, shall not be deemed to be held under
     any trust for the benefit of the Director or his beneficiaries or to be
     considered security for the performance of the obligations of the
     Association but shall be, and remain, a general, unpledged, unrestricted
     asset of the Association.

     D.  The Association hereby reserves the right to accelerate the payments
     specified in Sections 1, 2 and 3 above without the consent of the Director,
     his estate, beneficiaries, or any other person claiming through or under
     him.

     E.  The Association agrees that it will not merge or consolidate with any
     other Association or organization, or permit its business activities to be
     taken over by any other organization unless and until the succeeding or
     continuing Association or other organization shall expressly assume the
     rights and obligations of the Association herein set forth.  The
     Association further agrees that it will not cease its business activities
     or terminate its existence, other than as set forth in this Section,
     without having made adequate provision for the fulfilling of its
     obligations hereunder.

     F.  This Agreement may be revoked or amended in whole or in part by a
     writing signed by both of the parties hereto.

     G.  This Agreement shall be subject to and construed under the laws of the
     State of North Carolina.
<PAGE>
 
     IN WITNESS THEREOF, the said Association has caused this Agreement to be
signed in its Corporate name by its duly authorized officer, and impressed with
the Corporate seal, attached by its Secretary, and the said Director has
hereunto set his hand and seal, all on the date and year first above written.

ATTEST:                         Haywood Savings and Loan Association



/s/ Johnnie Sue Buchanan        By: /s/ Robert H. Francis    (Seal)
------------------------           ------------------------        
                                   Chairman



WITNESS:Jay W. Hargelte            /s/ Larry R. Ammons       (Seal)
        ---------------            ------------------------      
                                   The Director
<PAGE>
 
                        HAYWOOD SAVINGS BANK, INC., SSB
                          RETIREMENT PAYMENT AGREEMENT
                        _______________________________

                               1998 Amendment to
                         Agreement dated August 1, 1987
                        _______________________________

     WHEREAS, Haywood Savings Bank, Inc., SSB (the "Bank") entered into a
Retirement Payment Agreement (the "RPA") dated August 1, 1987, with
_____________________ (the "Director"); and

     WHEREAS, the Bank's Board of Directors and the Director have determined
that it is in their respective best interests to amend the RPA in the manner set
forth herein.

     NOW, THEREFORE, the RPA shall be amended as follows, pursuant to 
Section 5.F thereof:

     1.   Section 5 of the RPA shall be amended by adding the following new
paragraph H at the end thereof:

          H.  Notwithstanding the provisions of paragraphs 2, 3, and 4 hereof,
          any benefits that are or become payable after the closing date of a
          Change in Control (as defined in the Bank's Retirement Plan for Non-
          Employee Directors) shall be paid in the manner selected by the
          Director in a duly executed election in the form (the "Election Form")
          attached hereto as Exhibit "A"; provided that such an election will be
          honored and given effect only if it is properly made and delivered to
          the Bank more than 90 days before said closing date.  Present value
          determinations, as well as interest accruals on present value sums
          that are paid in installments over a fixed period of years, shall be
          calculated at a rate equal to 120% of the applicable federal rate,
          compounded semiannually, as determined under Section 1274(d) of the
          Internal Revenue Code of 1986, as amended, and the regulations
          thereunder.

               The Director may specify on the Election Form, the manner of
          payment to his beneficiary, and may at any time or from time to time
          change the identity or manner of payment to his beneficiary.

     2.   Nothing contained herein shall be held to alter, vary, or affect any
of the terms, provisions, or conditions of the RPA other than stated above.
<PAGE>
 
     WHEREFORE, the undersigned hereby approve this 1998 Amendment to the RPA.

Date of Execution:  ____________________________, 1998

ATTEST:                       HAYWOOD SAVINGS BANK, INC., SSB



_______________________            By_____________________________

                                   Its____________________________



WITNESS:                      DIRECTOR



_______________________       _____________________________________
<PAGE>
 
                          RETIREMENT PAYMENT AGREEMENT

AGREEMENT ENTERED INTO AS OF THE 1ST DAY OF April, 1991 between Haywood Savings
                                 ---        -----------         ---------------
and Loan Association, a domestic Corporation having its principal office in
--------------------                                                       
Waynesville, North Carolina (hereinafter referred to as the Association) and
-----------  --------------                                                 
Larry R. Ammons of Waynesville, North Carolina (hereinafter referred to as the
---------------    -----------  --------------                                
Director).

WITNESSETH:

WHEREAS, the Director is rendering valuable service and it is the desire of the
Association to have the benefit of his continued loyalty and service and also to
assist him in providing for the contingencies of retirement and death; and,

WHEREAS, the Director hereby agrees to waive fees paid to him as a Director in
the amount of $125.00 per month for five years from the date of the execution of
               ------               ----                                        
this agreement;

NOW THEREFORE, in consideration of the premises contained herein, the parties
hereto mutually agree as follows:

     1.   Retirement Benefit:  Should the Director still be in the Directorship
          ------------------                                                   
of the Association upon attainment of his 65th birthday, the Association will
                                          ----                               
commence to pay him $846 per month for a continuous period of 120 months.  In
                    ----                                      ---            
the event that the Director should die after becoming entitled to receive said
monthly installments but before any or all of said installments have been paid,
the Association will pay or will continue to pay said installments to such
beneficiary or beneficiaries as the Director has directed by filing with the
Association a notice in writing.  In the event of the death of the last named
beneficiary before all the unpaid payments have been made, the balance of any
amount which remains unpaid at said death shall be paid in a single sum to the
executor or administrator of the estate of the last named beneficiary to die.
In the absence of any such beneficiary designation, any amount remaining unpaid
at the Director's death shall be commuted on the basis of eight percent per
annum compound interest and shall be paid in a single sum to the executor or
administrator of the Director's estate.

     2.   Death Benefit:  Should the Director die while in the Directorship of
          -------------                                                       
the Association and prior to the attainment of his 65th birthday, the
                                                   ----              
Association (beginning at a date to be determined by the Association but within
six months from the date of such death) will commence to pay $846 per month for
                                                             ----              
a continuous period of 120 months to such beneficiary or beneficiaries as the
                       ---                                                   
Director has directed by filing with the Association a notice in writing.
Irrespective of the above, however, if the Director dies as a result of suicide
within two years of the execution of this agreement, the death benefit shall not
exceed an amount equal to his waived Directors' fees plus interest at the rate
of eight percent per annum compounded annually.  In the event of the death of
the last named beneficiary before all the unpaid payments have been made, the
balance of any amount which remains unpaid at said death shall be commuted on
the basis of eight percent per annum compound interest and shall be paid in a
single sum to the executor or administrator of the estate of the last named
beneficiary to die.  In the absence of any such beneficiary designation, any
amount remaining unpaid at the Director's death shall be commuted on the basis
of eight percent per annum compound interest and shall be paid in a single sum
to the executor or administrator of the Director's estate.
<PAGE>
 
3.   Termination of Directorship:
     --------------------------- 

     A.  If the Director terminates his Directorship, for reasons other than
     death or the attainment of his 65th birthday, prior to two years from the
                                    ----                                      
     execution date of the Agreement, the Director's benefits shall be limited
     to his waived Director fees plus interest at the rate of eight percent per
     annum compounded annually and shall be paid in a single sum as soon as
     practical following the termination of his Directorship.

     B.  If the Director terminates his Directorship, for reasons other than
     death or the attainment of his 65th birthday, at the end of two years or
                                    ----                                     
     more from the execution date of this Agreement, he or his beneficiary, as
     applicable, shall be entitled upon the attainment of his 65th birthday, or
                                                              ----             
     his prior death, to a percentage of the retirement benefits stated in
     Section 1 of this Agreement as determined by the following table:

<TABLE>
<CAPTION> 

    FULL NUMBER OF YEARS SERVED        PERCENTAGE OF RETIREMENT
     AS DIRECTOR FROM DATE OF         BENEFITS STATED IN SECTION
    EXECUTION OF THIS AGREEMENT      1 OF THIS AGREEMENT TO WHICH
 UNTIL TERMINATION OF DIRECTORSHIP     THE DIRECTOR IS ENTITLED
-----------------------------------  ----------------------------
<S>                                  <C>
                                                                  
                                                                  
                 2                                40%             
                                                                  
                 3                                60%             
                                                                  
                 4                                80%             
                                                    
                 5                               100%
</TABLE>

4.   Forfeiture Provisions:
     --------------------- 

     A.  During the period the retirement benefit is payable to the Director
     under Section 1 of this Agreement, the Director shall not engage in
     business activities which are in competition with the Association without
     first obtaining the written consent of the Association.

     B.  During the period the retirement payment is payable to the Director
     under Section 1 of the Agreement, the Director shall be available to render
     consulting services to the Association upon request by an officer of the
     Association, but such requests shall not be made  more frequently than once
     each moth.  The Director shall not be considered to have breached this
     condition if he is unable to consult because of his mental or physical
     disability.

     C.  Payment of the retirement benefit under this Agreement may be
     terminated by the Association, if the Director fails to comply with either
     of the conditions set forth in paragraph (A) and (B) of this Section 4.
<PAGE>
 
5.   General Provisions:
     ------------------ 

     A. Except as otherwise provided by the Agreement, it is agreed that neither
     the Director, nor his beneficiary shall have any right to commute, sell,
     assign, transfer or otherwise convey the right to receive any payments
     hereunder, which payments and the right hereto are expressly declared to be
     nonassignable and nontransferable.

     B.  The benefits payable under the Agreement shall be independent of, and
     in addition to, any other employment agreements that may exist from time to
     time between the parties hereto, concerning any other compensation payable
     by the Association to the Director whether as salary, bonus, or otherwise.
     This Agreement shall not be deemed to constitute a contract of employment
     between the parties hereto, nor shall any provision hereof restrict the
     right of the Association to discharge the Director or restrict the right of
     the Director to terminate his Directorship.

     C.  The rights of the Director under this Agreement and of any beneficiary
     of the Director shall be solely those of an unsecured creditor of the
     Association.  Any asset acquired by the Association in connection with the
     liabilities assumed by it hereunder, shall not be deemed to be held under
     any trust for the benefit of the Director or his beneficiaries or to be
     considered security for the performance of the obligations of the
     Association but shall be, and remain, a general, unpledged, unrestricted
     asset of the Association.

     D.  The Association hereby reserves the right to accelerate the payments
     specified in Sections 1, 2 and 3 above without the consent of the Director,
     his estate, beneficiaries, or any other person claiming through or under
     him.

     E.  The Association agrees that it will not merge or consolidate with any
     other Association or organization, or permit its business activities to be
     taken over by any other organization unless and until the succeeding or
     continuing Association or other organization shall expressly assume the
     rights and obligations of the Association herein set forth.  The
     Association further agrees that it will not cease its business activities
     or terminate its existence, other than as set forth in this Section,
     without having made adequate provision for the fulfilling of its
     obligations hereunder.

     F.  This Agreement may be revoked or amended in whole or in part by a
     writing signed by both of the parties hereto.

     G.  This Agreement shall be subject to and construed under the laws of the
     State of North Carolina.
<PAGE>
 
     IN WITNESS THEREOF, the said Association has caused this Agreement to be
signed in its Corporate name by its duly authorized officer, and impressed with
the Corporate seal, attached by its Secretary, and the said Director has
hereunto set his hand and seal, all on the date and year first above written.

ATTEST:   HAYWOOD SAVINGS AND LOAN ASSOCIATION, INC. (Association)
          ------------------------------------------              



          BY: /s/ Robert H. Francis                  (Seal) 
             ---------------------------------------
                            Chairman



              /s/ Larry R. Ammons                    (Seal) 
             --------------------------------------- 
                          The Director



WITNESS: _______________________________________________
<PAGE>
 
                          BENEFICIARY DESIGNATION FORM

As Beneficiary to receive any death benefits payable on my behalf from Haywood
                                                                       -------
Savings and Loan Association incorporated in Waynesville, North Carolina, I
----------------------------                 -----------  --------------   
designate the following:


          NAME           DATE OF BIRTH         ADDRESS          RELATIONSHIP
          ----           -------------         -------          ------------

PRIMARY
-------
                                            501 Blink Bonny
Cristina C. Ammons           8/10/42        Waynesville, N.C. 28786    wife 
------------------------------------------------------------------------------ 

------------------------------------------------------------------------------ 
------------------------------------------------------------------------------ 

CONTINGENT, If Any:
----------         

First Union National Bank, Trustee
------------------------------------------------------------------------------- 
------------------------------------------------------------------------------- 

(Note: If more than one primary beneficiary is named, the benefit will be paid
in equal shares to those living.  Should the contingent beneficiaries be
eligible to receive the benefits, such benefits will be paid in equal shares to
such living contingent beneficiaries.  If none of the designated beneficiaries
are living at such time as the death benefit is payable, such benefit will be
paid to the Executor or Administrator of your Estate).

Name of Spouse if not given above:
                                    -------------------------------------------



                                    /s/ Larry R. Ammons
---------------------------------  ---------------------------------------------
          Witness                   Signature of Director



                                               26 June 1991
                                   ---------------------------------------------
                                                   Date


Note:  The original should be retained by the Association, one copy by the
     Director, and one copy forwarded to Smith/Broadhurst, Inc.
<PAGE>
 
                        HAYWOOD SAVINGS BANK, INC., SSB
                          RETIREMENT PAYMENT AGREEMENT
                        _______________________________

                                 1998 Amendment
                        to Agreement dated April 1, 1991
                        _______________________________

     WHEREAS, Haywood Savings Bank, Inc., SSB (the "Bank") entered into a
Retirement Payment Agreement (the "RPA") dated April 1, 1991, with
_____________________ (the "Director"); and

     WHEREAS, the Bank's Board of Directors and the Director have determined
that it is in their respective best interests to amend the RPA in the manner set
forth herein.

     NOW, THEREFORE, the RPA shall be amended as follows, pursuant to 
Section 5. F thereof:

     1.   Section 5 of  the RPA shall be amended by adding the following new
paragraph H at the end thereof:

          H.  Notwithstanding the provisions of paragraphs 2, 3, and 4 hereof,
          any benefits that are or become payable after the closing date of a
          Change in Control (as defined in the Bank's Retirement Plan for Non-
          Employee Directors) shall be paid in the manner selected by the
          Director in a duly executed election in the form (the "Election Form")
          attached hereto as Exhibit "A"; provided that such an election will be
          honored and given effect only if it is properly made and delivered to
          the Bank more than 90 days before said closing date.  Present value
          determinations, as well as interest accruals on present value sums
          that are paid in installments over a fixed period of years, shall be
          calculated at a rate equal to 120% of the applicable federal rate,
          compounded semiannually, as determined under Section 1274(d) of the
          Internal Revenue Code of 1986, as amended, and the regulations
          thereunder.

               The Director may specify on the Election Form, the manner of
          payment to his beneficiary, and may at any time or from time to time
          change the identity or manner of payment to his beneficiary.

     2.   Nothing contained herein shall be held to alter, vary, or affect any
of the terms, provisions, or conditions of the RPA other than stated above.
<PAGE>
 
Retirement Payment Amendment
1998 Amendment
Page 2


     WHEREFORE, the undersigned hereby approve this 1998 Amendment to the RPA.

Date of Execution:  ____________________________, 1998

ATTEST:                       HAYWOOD SAVINGS BANK, INC., SSB


_________________________     By_________________________________
                                Its______________________________



WITNESS:                      DIRECTOR


________________________      ___________________________________