<PAGE>
 

                                                                    EXHIBIT 10.4
                            SEVERANCE PAY AGREEMENT
                            -----------------------


     THIS SEVERANCE PAY AGREEMENT ("Agreement") is made and entered into as of
the 1st day of July, 1998, by and among PREMIER BANK ("Premier"), THE BANK OF
GWINNETT COUNTY ("Gwinnett") (Premier and Gwinnett shall sometimes be
collectively referred to as "Employers"), and JOHN C. PENTECOST ("Employee") and
shall become effective only upon the closing of that certain Agreement and Plan
of Reorganization, dated as of February 5, 1998, by and between Button Gwinnett
Financial Corporation ("Button Gwinnett") and the Holding Company (the
"Effective Date").

                              W I T N E S S E T H:

     WHEREAS, Premier is a wholly owned subsidiary of  Premier Bancshares, Inc.
(the "Holding Company");

     WHEREAS, as of the Effective Date, Button Gwinnett, a Georgia corporation,
merged with and into the Holding Company, and Gwinnett became a wholly-owned
banking subsidiary of the Holding Company;

     WHEREAS, as of the Effective Date, Employee became Senior Vice President of
Commercial Lending of Premier and continued as Senior Vice President of
Commercial Lending of Gwinnett;

     WHEREAS, the Boards of Directors of Employers consider the establishment
and maintenance of highly competent and skilled management personnel for
Employers to be essential to protecting and enhancing the best interests of
Employers;

     WHEREAS, the Boards of Directors of Employers are desirous of inducing
Employee to remain in the employ of Employers, subject to the terms and
conditions hereof; and
<PAGE>
 
     WHEREAS, the parties agree that the provisions of this Agreement shall
control with respect to the rights and obligations of the parties resulting from
the future termination of the employment of Employee by Employers;

     NOW, THEREFORE, for and in consideration of the mutual covenants herein
contained, and other good and valuable consideration, the receipt and
sufficiency of which are hereby acknowledged, the parties hereto do hereby agree
as follows:

     1.  Definition of Change of Control.  "Change of Control" shall be deemed
         -------------------------------                                      
to have occurred:

          (a) Upon the consummation of any transaction in which any person (or
     persons acting in concert), partnership, corporation, or other organization
     shall own, control, or hold with the power to vote more than fifty percent
     (50%) of any class of voting securities of the Holding Company;
      --                                                            

          (b) Upon the consummation of any transaction in which the Holding
     Company, or substantially all of the assets of the Holding Company, shall
     be sold or transferred to, or consolidated or merged with, another
     corporation; or

          (c) Upon the consummation of any transaction in which either or both
     Employers, or substantially all of the assets of either or both Employers,
     shall be sold or transferred to, or consolidated or merged with, another
     corporation which is not a majority owned subsidiary of the Holding
     Company;

provided, however, if either or both Premier and Gwinnett shall become a
subsidiary of another corporation or shall be merged or consolidated into
another corporation and a majority of the outstanding voting shares of the new
parent or surviving corporation are owned, immediately after such acquisition,
merger, or consolidation, by the owners of a majority of the voting shares of
the Holding Company immediately before such acquisition, merger, or
consolidation, then no Change

                                      -2-
<PAGE>
 
of Control shall be deemed to have occurred.  It is contemplated by the parties
that, during the Term of Employment, Gwinnett may be merged with and into
Premier; and further that, upon such merger, Premier, for all purposes under
this Agreement, shall be deemed the sole Employer of Employee and referred to as
"Employers."

     2.   Employment.  Employee agrees to remain in the employ of Employers as
          ----------                                                          
an employee at will under Georgia law.  Employee agrees to perform faithfully
such services as are reasonably consistent with his position(s) and shall from
time to time be assigned to him by the Boards of Directors of Employers in a
trustworthy and businesslike manner for the purpose of advancing the interests
of  Employers.  The Boards of Directors of Employer may also from time to time
change Employee's position or alter his duties and responsibilities and assign a
new position or new duties and responsibilities that are similar in scope and
nature to Employee's existing position, duties, and responsibilities without
invalidating this Agreement or effecting the termination of Employee.  At all
times, Employee shall manage and conduct the business of Employers in accordance
with the policies established by the Boards of Directors of Employers and in
compliance with applicable regulations promulgated by governing regulatory
agencies.  Responsibility for the supervision of Employee shall rest with the
Chief Executive Officer(s) of Employers.  The Boards of Directors of Employers
shall have the authority to terminate Employee, subject only to the provisions
outlined in paragraph 4 of this Agreement.

     3.   Performance of Duties.  During the period of employment hereunder,
          ---------------------                                             
except for periods of illness, disability, reasonable vacation periods, and
reasonable leaves of absence, Employee shall devote substantially all of his
business time, attention, skill, and efforts to the faithful performance of his
duties hereunder.  Employee shall be entitled to reasonable participation as a
member in community, civic, or similar organizations and the pursuit of personal
investments which do not present any material conflict of interest with
Employers, or otherwise unfavorably

                                      -3-
<PAGE>
 
affect the performance of Employee's duties pursuant to this Agreement.  The
Employee shall have no employment contract or other written or oral agreement
concerning employment with any entity or person other than Employers during the
term of his employment under this Agreement.

     4.   Payments to Employee Upon Termination of Employment.  The Boards of
          ---------------------------------------------------                
Directors of Employers may terminate Employee's employment under this Agreement
at any time.  Employee may voluntarily terminate his employment under this
Agreement.  The rights and obligations of Employers and Employee in the event of
such termination are set forth in this paragraph 4 as follows:

          (a) Termination Without Change of Control.  Employee shall have no
              -------------------------------------                         
     right to compensation or other benefits for any period after the
     termination of his employment, other than a termination after a Change of
     Control, regardless of whether the termination is with or without cause or
     the termination is voluntary or involuntary.

          (b) Event of Termination in Connection With a Change of Control.  If,
              -----------------------------------------------------------      
     within six (6) months immediately following a Change of Control or within
                 -                                                            
     three (3) months immediately prior to such Change of Control, Employee's
            -                                                                
     employment with Employers is terminated by Employers or by Employee, then
     the Employers shall pay to Employee, or in the event of his subsequent
     death, his designated beneficiary or beneficiaries, or his estate, as the
     case may be, shall receive, as liquidated damages, in lieu of all other
     claims, a severance payment equal to the sum of Employee's then current
     annual salary plus any incentive compensation paid to Employee during the
     immediately preceding twelve (12) months, to be paid in full on the last
                                   --                                        
     day of the month following the date of said termination. The responsibility
     for making such severance payment shall be allocated among Employers in
     whatever manner they deem appropriate.  In no event shall the total
     compensation paid to Employee upon the termination of his employment in
     connection with a Change of Control

                                      -4-
<PAGE>
 
     exceed the amount permitted by Section 280G of the Internal Revenue Code
     (as amended) or three times Employee's average annual compensation.

     5.   Regulatory Suspension.
          --------------------- 

          (a) If Employee is suspended and/or temporarily prohibited from
     participating in the conduct of the affairs of Employers by a notice served
     under Sections 8(e)(3) or (g)(1) of the Federal Deposit Insurance Act, 12
     U.S.C. (S)1818(e)(3) or (g)(1), then the obligations of Employers under
     this Agreement shall be suspended as of the date of service of such notice,
     unless stayed by appropriate proceedings.  If the charges in the notice are
     dismissed, then the Employers may, in their discretion, (i) pay Employee
     all or part of the compensation withheld while its contract obligations
     were suspended and (ii) reinstate in whole or in part any of its
     obligations which were suspended.

          (b) If  Employee is removed and/or permanently prohibited from
     participating in the conduct of the affairs of Employer by an order issued
     under Section 8(e)(4) or (g)(1) of the Federal Deposit Insurance Act, 12
     U.S.C. (S) 1818(e)(4) or (g)(1), then all obligations of the Employers
     under this Agreement shall terminate as of the effective date of the order.

     6.   Source of Payments.  All payments provided herein shall be paid in
          ------------------                                                
cash from the general funds of Employers as provided herein; no special or
separate fund shall be established by Employers; and no other segregation of
assets shall be made to assure payment.  Employee shall have no right, title, or
interest in or to any investments which Employers may make to meet the
obligations hereunder.

     7.   Nondisclosure of Confidential Information.  Employee acknowledges that
          -----------------------------------------                             
he possesses confidential information of a special and unique nature and value
affecting and relating to both Employers' and the Holding Company's business,
including, without limitation, customer lists, deposits, business records, other
trade secrets, and other similar confidential information

                                      -5-
<PAGE>
 
relating to Employers and/or the Holding Company and the business of each (all
the foregoing being hereinafter collectively referred to as "Confidential
Information").  Employee recognizes and acknowledges that all Confidential
Information is the exclusive property of Employers and/or the Holding Company,
respectively, constitutes trade secrets of Employers and/or the Holding Company,
is material and confidential, and greatly affects the goodwill and the effective
and successful conduct of the business of Employers and/or the Holding Company.
As a material inducement to Employers to enter into this Agreement and to employ
Employee, Employee covenants and agrees that he will not at any time during the
term of his employment under this Agreement, and for a period of one (1) year
                                                                      -      
from the end of such employment, directly or indirectly, divulge, reveal, or
communicate any Confidential Information to any person, firm, corporation, or
entity whatsoever, or use any Confidential Information for his own benefit or
for the benefit of others.  Employee further acknowledges that said Confidential
Information has material commercial value to Employers and/or the Holding
Company so long as it is not known by competitors of Employers and/or the
Holding Company and that both the Employers and the Holding Company have taken
reasonable steps to keep all such information and trade secrets confidential.

     8.   Injunctions.  In view of the irreparable harm and damage which
          -----------                                                   
Employers and/or the Holding Company would sustain as a result of a breach by
Employee of the covenants or agreements under paragraph 7 hereof, and in view of
the lack of an adequate remedy at law to protect the interests of Employers
and/or the Holding Company, Employers and/or the Holding Company shall have the
right to receive, and Employee hereby consents to the issuance of, a permanent
injunction enjoining Employee from any violation of the covenants and agreements
set forth in paragraph 7 hereof.  The foregoing remedy shall be in addition to,
and not in limitation of, any other rights or remedies to which Employers and/or
the Holding Company is/are or may be entitled at law or in equity respecting
this Agreement.  It is expressly agreed by the parties hereto that the Holding

                                      -6-
<PAGE>
 
Company is an intended third party beneficiary of paragraphs 7, 8, and 9 of this
Agreement and may enforce same against Employee as if it were a party hereto.

     9.   Attorneys' Fees.  In the event any party hereto is required to engage
          ---------------                                                      
in legal action against any other party hereto, either as plaintiff or
defendant, in order to enforce or defend any of its or his rights under this
Agreement, and such action results in a final judgment in favor of one or more
parties, then the party or parties against whom said final judgment is obtained
shall reimburse the prevailing party or parties for all legal fees and expenses
incurred by the prevailing party or parties in asserting or defending its or his
rights hereunder.

     10.  Effect of Prior Agreements.  This Agreement contains the entire
          --------------------------                                     
understanding between the parties hereto and supersedes any prior agreements
relating to employment and any contemporaneous oral agreement or understanding
by, between, or among the Employers and Employee.

     11.  General Provisions.
          ------------------ 

          (a) Nonassignability.  Neither this Agreement nor any right or
              ----------------                                          
     interest hereunder shall be assignable by Employee, his beneficiaries or
     legal representatives, without the written consent of Employers; provided,
     however, that nothing in this paragraph 11(a) shall preclude (i) Employee
     from designating a beneficiary to receive any benefits payable hereunder
     upon his death, or (ii) the executors, administrators, or other legal
     representatives of Employee or his estate from assigning any rights
     hereunder to the person or persons entitled thereto.

          (b) No Attachment.  Except as required by law, no right to receive
              -------------                                                 
     payments under this Agreement shall be subject to anticipation,
     commutation, alienation, sale, assignment, encumbrance, charge, pledge, or
     hypothecation, or to execution, attachment,

                                      -7-
<PAGE>
 
     levy, and any attempt, voluntary or involuntary, to effect any such action
     shall be null, void, and of no effect.

          (c) Binding Agreement.  This Agreement shall be binding upon, and
              -----------------                                            
     inure to the benefit of, the Employers and Employee and their respective
     heirs, successors, assigns, and legal representatives.

     12.  Modification and Waiver.
          ----------------------- 

          (a) Amendment of Agreement.  This Agreement may not be modified or
              ----------------------                                        
     amended except by an instrument in writing, signed by the parties hereto,
     and which specifically refers to this Agreement.

          (b) Waiver.  No term or condition of this Agreement shall be deemed to
              ------                                                            
     have been waived, nor shall there be any estoppel against the enforcement
     of any provision of this Agreement, except by written instrument of the
     party charged with such waiver or estoppel. No such written waiver shall be
     deemed a continuing waiver unless specifically stated therein, and each
     waiver shall operate only as to the specific term or condition waived and
     shall not constitute a waiver of such term or condition for the future or
     as to any act other than that specifically waived.

     13.  Severability.  If for any reason any provision of this Agreement is
          ------------                                                       
held invalid, such invalidity shall not affect any other provision of this
Agreement not held invalid, and each such other provision shall to the full
extent consistent with law continue in full force and effect.  If any provision
of this Agreement shall be held invalid in part, such invalidity shall in no way
affect the rest of such provision not held so invalid, and the rest of such
provision, together with all other provisions of this Agreement, shall to the
full extent consistent with law continue in full force and effect.

                                      -8-
<PAGE>
 
     14.  Headings.  The headings of paragraphs herein are included solely for
          --------                                                            
convenience of reference and shall not control the meaning or interpretation of
any of the provisions of this Agreement.

     15.  Governing Law.  This Agreement has been executed and delivered in the
          -------------                                                        
State of Georgia, and its validity, interpretation, performance, and enforcement
shall be governed by the laws of said State.

     16.  Rights of Third Parties.  Nothing herein expressed or implied is
          -----------------------                                         
intended to or shall be construed to confer upon or give to any person, firm, or
other entity, other than the parties hereto and their permitted assigns, any
rights or remedies under or by reason of this Agreement, except as expressly set
forth in paragraph 8 hereof.

     17.  Notices.  All notices, requests, demands, and other communications
          -------                                                           
provided for by this Agreement shall be in writing and shall be sufficiently
given if and when mailed in the United States by registered or certified mail,
or personally delivered, to the party entitled thereto at the address stated
below or to such changed address as the addressee may have given by a similar
notice:

          To Employers:  Chairman, Compensation Committee
                         Board of Directors
                         Premier Bancshares, Inc.
                         2180 Atlanta Plaza
                         950 E. Paces Ferry Road
                         Atlanta, Georgia  30326


          Copy to:       Steven S. Dunlevie, Esq.
                         Womble Carlyle Sandridge & Rice, PLLC
                         Suite 700
                         1275 Peachtree Street
                         Atlanta, Georgia 30309


          To Employee:   Mr. John C. Pentecost
                         584 Maplewood Drive
                         Lawrenceville, Georgia 30045

                                      -9-
<PAGE>
 
          IN WITNESS WHEREOF, Employers have caused this Agreement to be
executed and its seal to be affixed hereunto by their duly authorized officers,
and Employee has signed this Agreement under seal, as of the Effective Date.

ATTEST:                                   PREMIER BANK


/s/                                       By:  /s/ Darrell D. Pittard   
---------------------------                   -------------------------------
Secretary                                      Darrell D. Pittard
(BANK SEAL)                                    Chairman of the Board



ATTEST:                                   THE BANK OF GWINNETT COUNTY


/s/                                       By:  /s/ Glenn S. White   
--------------------------                    -------------------------------
Secretary                                      Glenn S. White
(BANK SEAL)                                    President



/s/                                        /s/ John C. Pentecost   
--------------------------                -----------------------------(SEAL)
Witness                                    John C. Pentecost

                                      -10-