<PAGE>   1

                                                                 EXHIBIT 10.4


                           FIRST FEDERAL OF THE SOUTH

                              EMPLOYMENT AGREEMENT

                                OCTOBER 1, 1997

                               (JOE K. MCARTHUR)



                 THIS AGREEMENT is entered into as of the 1st day of October,
1997 (the "Effective Date"), by and between First Federal of the South (the
"Association") and Joe K. McArthur (the "Employee").

                 WHEREAS, the Employee has heretofore been employed by the
Association as Executive Vice-President and Chief Operating Officer and is
experienced in all phases of the business of the Association; and

                 WHEREAS, the parties desire by this writing to establish and
to set forth the employment relationship between the Association and the
Employee.

                 NOW, THEREFORE, it is AGREED as follows:

                 1.       Employment.  The Employee is hereby employed as the
Executive Vice-President and Chief Operating Officer of the Association.  The
Employee shall render such administrative and management services for the
Association as are currently rendered and as are customarily performed by
persons situated in a similar executive capacity.  The Employee shall also
promote, by entertainment or otherwise, as and to the extent permitted by law,
the business of the Association.  The Employee's other duties shall be such as
the Board of Directors of the Association ("Board") may from time to time
reasonably direct, including normal duties as an officer of the Association.

                 2.       Base Compensation.  The Association agrees to pay the
Employee during the term of this Agreement a salary at the rate of $105,000 per
annum, payable in cash not less frequently than monthly.  The Board shall
review, not less often than annually, the rate of the Employee's salary, and in
its sole discretion may decide to increase his salary.

                 3.       Discretionary Bonuses.  The Employee shall
participate in an equitable manner with all other senior management employees
of the Association in discretionary bonuses that the Board may award from time
to time to the Association's senior management employees.  No other
compensation provided for in this Agreement shall be deemed a substitute for
the Employee's right to participate in such discretionary bonuses.

                 4.       (a)     Participation in Retirement, Medical and
Other Plans.  The Employee shall participate in any plan that the Association
maintains for the benefit of its employees if the plan relates to (i) pension,
profit-sharing, or other retirement benefits, (ii) medical insurance or the
reimbursement of medical or dependent care expenses, or (iii) other group
benefits, including disability and life insurance plans.

                          (b)     Employee Benefits; Expenses.  The Employee
shall participate in any fringe benefits which are or may become available to
the Association's senior management employees, including for example: any stock
option or incentive compensation plans, club memberships, and any other
benefits

                                      1
<PAGE>   2


which are commensurate with the responsibilities and functions to be performed
by the Employee under this Agreement.  The Employee shall be reimbursed for all
reasonable out-of-pocket business expenses which he shall incur in connection
with his services under this Agreement upon substantiation of such expenses in
accordance with the policies of the Association.

                 5.       Term.  The Association hereby employs the Employee,
and the Employee hereby accepts such employment under this Agreement, for the
period commencing on the Effective Date and ending 36 months thereafter (or
such earlier date as is determined in accordance with Section 9).
Additionally, on each annual anniversary date from the Effective Date, this
Agreement and the Employee's term of employment shall be extended for an
additional one-year period beyond the then effective expiration date, provided
the Board determines in a duly adopted resolution that the performance of the
Employee has met the Board's requirements and standards, and that this
Agreement shall be extended.

                 6.       Loyalty; Full Time and Attention.

                          (a)     During the period of his employment hereunder
and except for illnesses, reasonable vacation periods, and reasonable leaves of
absence, the Employee shall devote all his full business time, attention,
skill, and efforts to the faithful performance of his duties hereunder;
provided, however, from time to time, Employee may serve on the boards of
directors of, and hold any other offices or positions in, companies or
organizations, which will not present any conflict of interest with the
Association or any of its subsidiaries or affiliates, or unfavorably affect the
performance of Employee's duties pursuant to this Agreement, or will not
violate any applicable statute or regulation.  "Full business time" is hereby
defined as that amount of time usually devoted to like companies by similarly
situated executive officers.  During the term of his employment under this
Agreement, the Employee shall not engage in any business or activity contrary
to the business affairs or interests of the Association, or be gainfully
employed in any other position or job other than as provided above.

                          (b)     Nothing contained in this Paragraph 6 shall
be deemed to prevent or limit the Employee's right to invest in the capital
stock or other securities of any business dissimilar from that of the
Association, or, solely as a passive or minority investor, in any business.

                 7.       Standards.  The Employee shall perform his duties
under this Agreement in accordance with such reasonable standards as the Board
may establish from time to time.  The Association will provide Employee with
the working facilities and staff customary for similar executives and necessary
for him to perform his duties.

                 8.       Vacation and Sick Leave.  The Employee shall be
entitled, without loss of pay, to absent himself voluntarily from the
performance of his duties under this Agreement in accordance with the terms set
forth below, all such voluntary absences to count as vacation time; provided
that:

                          (a)     The Employee shall be entitled to an annual
vacation in accordance with the policies that the Board periodically
establishes for senior management employees of the Association.

                          (b)     The Employee shall not receive any additional
compensation from the Association on account of his failure to take a vacation,
and the Employee shall not accumulate unused vacation from one fiscal year to
the next, except in either case to the extent authorized by the Board.

                          (c)     In addition to the aforesaid paid vacations,
the Employee shall be entitled without loss of pay, to absent himself
voluntarily from the performance of his employment obligations with the
Association for such additional periods of time and for such valid and
legitimate reasons as the Board may

                                       2
<PAGE>   3


in its discretion approve.  Further, the Board may grant to the Employee a
leave or leaves of absence, with or without pay, at such time or times and upon
such terms and conditions as the Board in its discretion may determine.

                          (d)     In addition, the Employee shall be entitled
to an annual sick leave benefit as established by the Board.

                 9.       Termination and Termination Pay.  Subject to the
provisions of Section 11 hereof, the Employee's employment hereunder may be
terminated under the following circumstances:

                          (a)     Death.  The Employee's employment under this
Agreement shall terminate upon his death during the term of this Agreement, in
which event the Employee's estate shall be entitled to receive the compensation
due the Employee through the last day of the calendar month in which his death
occurred.

                          (b)     Disability.  The Association may terminate
the Employee's employment after having established, through a determination by
the Board, the Employee's Disability.  For purposes of this Agreement,
"Disability" means a physical or mental infirmity which impairs the Employee's
ability to substantially perform his duties under this Agreement and which
results in the Employee becoming eligible for long-term disability benefits
under the Association's long-term disability plan (or, if the Association has
no such plan in effect, which impairs the Employee's ability to substantially
perform his duties under this Agreement for a period of one hundred eighty
(180) consecutive days).  The Employee shall be entitled to the compensation
and benefits provided for under this Agreement for (i) any period during the
term of this Agreement and prior to the establishment of the Employee's
Disability during which the Employee is unable to work due to the physical or
mental infirmity, or (ii) any period of Disability which is prior to the
Executive's termination of employment pursuant to this Section 9(b).

                          (c)     For Cause.  The Board may, by written notice
to the Employee, immediately terminate his employment at any time, for Cause.
The Employee shall have no right to receive compensation or other benefits for
any period after termination for Cause.  Termination for "Cause" shall mean
termination because of, in the good faith determination of the Board, the
Employee's personal dishonesty, incompetence, willful misconduct, breach of
fiduciary duty involving personal profit, intentional failure to perform stated
duties, willful violation of any law, rule or regulation (other than traffic
violations or similar offenses) or final cease-and-desist order, or material
breach of any provision of this Agreement.  Notwithstanding the foregoing, the
Employee shall not be deemed to have been terminated for Cause unless there
shall have been delivered to the Employee a copy of a resolution duly adopted
by the affirmative vote of not less than a majority of the entire membership of
the Board (excluding the Employee if a member of the Board) at a meeting of the
Board called and held for the purpose (after reasonable notice to the Employee
and an opportunity for the Employee to be heard before the Board), finding that
in the good faith opinion of the Board the Employee was guilty of conduct set
forth above in the second sentence of this Subsection (c) and specifying the
particulars thereof in detail.

                          (d)     Without Cause.  Subject to the provisions of
Section 11 hereof, the Board may, by written notice to the Employee,
immediately terminate his employment at any time for any reason; provided that
if such termination is for any reason other than pursuant to Sections 9 (a) (b)
or (c) above, the Employee shall be entitled to receive the following
compensation and benefits: (i) the salary provided pursuant to Section 2
hereof, up to the date of expiration of the term (including any renewal term
then in effect) of this Agreement (the "Termination Date"), plus said salary
for an additional 12-month period, and (ii) the cost to the Employee of
obtaining all health, life, disability and other benefits (excluding any bonus,
stock option or other compensation benefits) which the Employee would have been
eligible to participate in

                                       3
<PAGE>   4


through the Termination Date based upon the benefit levels substantially equal
to those that the Association provided for the Employee at the date of
termination of employment.  Said sum shall be paid, at the option of the
Employee, either (I) in periodic payments over the remaining term of this
Agreement, as if the Employee's employment had not been terminated, or (II) in
one lump sum within ten (10) days of such termination; provided however, that
the amount to be paid by the Association to the Employee hereunder shall not
exceed three (3) times the Employee's "average annual compensation".  The
Employee's "annual average compensation" shall be the average of the total
annual "compensation" acquired by the Employee during each of the five (5)
fiscal years (or the number of full fiscal years of employment, if the
Employee's employment is less than five (5) years at the termination thereof)
immediately preceding the date of termination.  The term "compensation" shall
mean any payment of money or provision of any other thing of value in
consideration of employment, including, without limitation, base compensation,
bonuses, pension and profit sharing plans, directors fees or committees fees,
fringe benefits and deferred compensation accruals.


                          (e)     Voluntary Termination by Employee.  Subject
to the provisions of Section 11 hereof, the Employee may voluntarily terminate
employment with the Association during the term of this Agreement, upon at
least 60 days' prior written notice to the Board of Directors, in which case
the Employee shall receive only his compensation, vested rights and employee
benefits accrued up to the date of his termination.

                 10.      No Mitigation.  The Employee shall not be required to
mitigate the amount of any payment provided for in this Agreement by seeking
other employment or otherwise, and no such payment shall be offset or reduced
by the amount of any compensation or benefits provided to the Employee in any
subsequent employment.

                 11.      Change in Control.

                          (a)     Notwithstanding any provision herein to the
contrary, if the Employee's employment under this Agreement is terminated by
the Association, without the Employee's prior written consent and for a reason
other than for Cause, death or disability in connection with or within
twenty-four (24) months after any change in control of the Association or
SouthFirst Bancshares, Inc. (the "Corporation"), the Employee shall be paid an
amount equal to the difference between (i) the product of 2.99 times his "base
amount" as defined in Section 28OG(b)(3) of the Internal Revenue Code of 1986,
as amended (the "Code") and regulations promulgated thereunder, and (ii) the
sum of any other "parachute payments" (as defined under Section 28OG(b)(2) of
the Code) that the Employee receives on account of the change in control.  Said
sum shall be paid in one lump sum within ten (10) days of such termination.
The term "change in control" shall mean (1) a change in the ownership, holding
or power to vote more than 25% of the Association's or corporation's voting
stock, (2) a change in the ownership or possession of the ability to control
the election of a majority of the Association's or Corporation's directors, (3)
a change in the ownership or possession of the ability to exercise a
controlling influence over the management or policies of the Association or the
Corporation by any person or by persons acting as a "group" (within the meaning
of Section 13(d) of the Securities Exchange Act of 1934) (except in the case of
(1), (2) and (3) hereof, ownership or control of the Association or its
directors by the Corporation itself shall not constitute a ("change in
control"), or (4) during any period of two consecutive years, individuals who
at the beginning of such period constitute the Board of Directors of the
Corporation or the Association (the "Company Board") (the "Continuing
Directors") cease for any reason to constitute at least two-thirds thereof,
provided that any individual whose election or nomination for election as a
member of the Company Board was approved by a vote of at least two-thirds of
the Continuing Directors then in office shall be considered a Continuing
Director.  The term "person" means an individual other than the Employee, or a
corporation, partnership, trust, association, joint venture, pool,

                                       4
<PAGE>   5

syndicate, sole proprietorship, unincorporated organization or any other form
of entity not specifically listed herein.

                          (b)     Notwithstanding any other provision of this
Agreement to the contrary, the Employee may voluntarily terminate his
employment under this Agreement within twelve (12) months following a change in
control of the Association or the Corporation, and the Employee shall thereupon
be entitled to receive the payment described in Section 11(a) of this
Agreement, upon the occurrence of any of the following events, or within ninety
(90) days thereafter, which have not been consented to in advance by the
Employee in writing:  (i) the requirement that the Employee move his personal
residence, or perform his principal executive functions, more than thirty-five
(35) miles from his primary office as of the date of the change in control;
(ii) a material reduction in the Employee's base compensation as in effect on
the date of the change in control, as the same may be increased from time to
time; (iii) the failure by the Association to continue to provide the Employee
with compensation and benefits provided for under this Agreement, as the same
may be increased from time to time, or with benefits substantially similar to
those provided to him under any of the employee benefit plans in which the
Employee now or hereafter becomes a participant, or the taking of any action by
the Association which would directly or indirectly reduce any of such benefits
or deprive the Employee of any material fringe benefit enjoyed by him at the
time of the change in control; (iv) the assignment to the Employee of duties
and responsibilities materially different from those normally associated with
his position as referenced at Section 1; (v) a failure to elect or reelect the
Employee to the Board of Directors of the Association if the Employee is
serving on such Board on the date of the change in control; or (vi) a material
diminution or reduction in the Employee's responsibilities or authority
(including reporting responsibilities) in connection with his employment with
the Association.

                          (c)     Any payments made to the Employee pursuant to
this Agreement, or otherwise, are subject to and conditioned upon their
compliance with 12 U.S.C. Section 1828(k) and any regulations promulgated
thereunder.

                          (d)     In the event that any dispute arises between
the Employee and the Association as to the terms or interpretation of this
Agreement, including this Section 11, whether instituted by formal legal
proceedings or otherwise, including an, action that the Employee takes to
enforce the terms of this Section 11 or to defend against any action taken by
the Association, the Employee shall be reimbursed for all costs and expenses,
including reasonable attorneys' fees, arising from such dispute, proceedings or
actions, provided that the Employee shall have obtained a final judgement by a
court of competent jurisdiction in favor of the Employee.  Such reimbursement
shall be paid within ten (10) days of Employee's furnishing to the Association
written evidence, which may be in the form, among other things, of a canceled
check or receipt, of any costs or expenses incurred by the Employee.

                 12.      Requirements of Applicable Regulations of OTS

                 (a)      The Association's board of directors may terminate
the Employee's employment at any time, but any termination by the Association's
board of directors, other than termination for cause, shall not prejudice the
Employee's right to compensation or other benefits under this Agreement.  The
Employee shall have no right to receive compensation or other benefits for any
period after termination for cause (as defined in Section 9(c) hereof).

                 (b)      If the Employee is suspended and/or temporarily
prohibited from participating in the conduct of the Association's affairs by a
notice served under section 8(e)(3) or (g)(1) of Federal Deposit Insurance Act
(12 U.S.C.  1818 (e)(3) and (g)(1)), the Association's obligations under this
Agreement shall be suspended as of the date of service unless stayed by
appropriate proceedings.  If the charges in the notice are dismissed, the
Association may in its discretion (i) pay the Employee all or part of the
compensation

                                       5
<PAGE>   6


withheld while its obligations were suspended, and (ii) reinstate (in whole or
in part) any of its obligations which were suspended.

                 (c)      If the Employee is removed and/or permanently
prohibited from participating in the conduct of the Association's affairs by an
order issued under section 8(e)(4) or (g)(1) of the Federal Deposit Insurance
Act (12 U.S.C. 1818 (e)(4) or (g)(1)), all obligations of the Association under
this Agreement shall terminate as of the effective date of the order, but
vested rights of the contracting parties shall not be affected.

                 (d)      If the Association is in default (as defined in
section 3(x)(1) of the Federal Deposit Insurance Act), all obligations under
this Agreement shall terminate as of the date of default, but this Section
12(d) shall not affect any vested rights of the parties.

                 (e)      All obligations under this Agreement shall be
terminated, except to the extent determined that the continuation of this
Agreement is necessary of the continued operation of the Association:

                          (i)     By the Director of the Office of Thrift
Supervision (the "Director") or his or her designee, at the time the Federal
Deposit Insurance Corporation or Resolution Trust Corporation enters into an
agreement to provide assistance to or on behalf of the Association under the
authority contained in 13(c) of the Federal Deposit Insurance Act; or

                          (ii)    By the Director or his or her designee, at
the time the Director or his or her designee approves a supervisory merger to
resolve problems related to operation of the association or when the
association is determined by the Director to be in an unsafe or unsound
condition.

                 Any rights of the Employee that have already vested, however,
shall not be affected by such action.

                 (f)      Should any provision of this Agreement give rise to a
discrepancy or conflict with respect to any applicable law or regulation, then
the applicable law or regulation shall control the relevant construction and
operation of this Agreement.

                                       6
<PAGE>   7



                 13.      Successors and Assigns.

                          (a)     This Agreement shall inure to the benefit of
and be binding upon any corporate or other successor of the Association which
shall acquire, directly or indirectly, by merger, consolidation, purchase or
otherwise, all or substantially all of the assets or stock of the Association.

                          (b)     since the Association is contracting for the
unique and personal skills of the Employee, the Employee shall be precluded
from assigning or delegating his rights or duties hereunder without first
obtaining the written consent of the Association.

                 14.      Amendments.  No amendments or additions to this
Agreement shall be binding unless made in writing and signed by all of the
parties, except as herein otherwise specifically provided.

                 15.      Applicable Law.  Except to the extent preempted by
Federal law, the laws of the State of Delaware shall govern this Agreement in
all respects, whether as to its validity, construction, capacity, performance
or otherwise.

                 16.      Severability.  The provisions of this Agreement shall
be deemed severable and the invalidity or unenforceability of any provision
shall not affect the validity or enforceability of the other provisions hereof.

                 17.      Entire Agreement.  This Agreement, together with any
understanding or modifications thereof as agreed to in writing by the parties,
shall constitute the entire agreement between the parties hereto.

                 IN WITNESS WHEREOF, the parties have executed this Agreement
on the day and year first hereinabove written.

ATTEST:                         FIRST FEDERAL OF THE SOUTH


                                BY: /s/ Donald C. Stroup
--------------------------          -----------------------------------------
Secretary                               President and Chief Executive Officer



WITNESS:

                                
                                 /s/ Joe K. McArthur
--------------------------      ----------------------------------------------
                                        Joe K. McArthur ("Employee")

                                       7