<PAGE>

                                                                   EXHIBIT 10.16

         SEVERANCE COMPENSATION AGREEMENT dated as of February 11, 2004, between
O'Charley's Inc., a Tennessee corporation (the "Company"), and A. Chad Fitzhugh
(the "Executive").

         The Company's Board of Directors has determined that it is appropriate
to reinforce and encourage the continued attention and dedication of certain
members of the Company's senior management, including the Executive, to their
assigned duties without distraction in potentially disturbing circumstances
arising from the possibility of a change in control of the Company.

         This Agreement sets forth the severance compensation which the Company
agrees it will pay to the Executive if the Executive's employment with the
Company terminates under one of the circumstances described herein following a
Change In Control of the Company (as defined herein).

         1.       TERM. This Agreement shall terminate, except to the extent
that any obligation of the Company hereunder remains unpaid as of such time,
upon the earliest of (i) three years from the date hereof if a Change in Control
of the Company has not occurred prior to such date; (ii) the termination of the
Executive's employment with the Company based on death, Disability (as defined
in Section 3(b)), Retirement (as defined in Section 3(c)) or Cause (as defined
in Section 3(d)) or by the Executive other than for Good Reason (as defined in
Section 3(e)); and (iii) eighteen months from the date of a Change in Control of
the Company if the Executive has not terminated his employment for Good Reason
as of such time.

         2.       CHANGE IN CONTROL. No compensation shall be payable under this
Agreement unless and until (a) there shall have been a Change in Control of the
Company while the Executive is still an employee of the Company and (b) the
Executive's employment by the Company thereafter shall have been terminated in
accordance with Section 3. For purposes of this Agreement, a Change in Control
means the happening of any of the following:

                  (i)      any person or entity, including a "group" as defined
         in Section 13(d)(3) of the Securities Exchange Act of 1934, other than
         the Company, a wholly-owned subsidiary thereof, any employee benefit
         plan of the Company or any of its Subsidiaries becomes the beneficial
         owner of the Company's securities having 30% or more of the combined
         voting power of the then outstanding securities of the Company that may
         be cast for the election of directors of the Company (other than as a
         result of an issuance of securities initiated by the Company in the
         ordinary course of business); or

                  (ii)     as the result of, or in connection with, any cash
         tender or exchange offer, merger or other business combination, sale of
         assets or contested election, or any combination of the foregoing
         transactions, less than a majority of the combined voting power of the
         then outstanding securities of the Company or any successor corporation
         or entity entitled to vote generally in the election of the directors
         of the Company or such other corporation or entity after such
         transaction are held in the aggregate by the holders of the Company's
         securities entitled to vote generally in the election of directors of
         the Company immediately prior to such transaction; or

<PAGE>

                  (iii)    during any period of two consecutive years,
         individuals who at the beginning of any such period constitute the
         Board cease for any reason to constitute at least a majority thereof,
         unless the election, or the nomination for election by the Company's
         shareholders, of each director of the Company first elected during such
         period was approved by a vote of at least two-thirds of the directors
         of the Company then still in office who were directors of the Company
         at the beginning of any such period.

         3.       TERMINATION FOLLOWING CHANGE IN CONTROL. (a) If a Change in
Control of the Company shall have occurred while the Executive is still an
employee of the Company, the Executive shall be entitled to the compensation
provided in Section 4 upon the subsequent termination of the Executive's
employment with the Company by the Executive or by the Company within eighteen
months of the Change in Control of the Company unless such termination is as a
result of (i) the Executive's death; (ii) the Executive's Disability (as defined
in Section (3)(b) below); (iii) the Executive's Retirement (as defined in
Section 3(c) below); (iv) the Executive's termination by the Company for Cause
(as defined in Section 3(d) below); or (v) the Executive's decision to terminate
employment other than for Good Reason (as defined in Section 3(e) below).

                  (b)      DISABILITY. If, as a result of the Executive's
incapacity due to physical or mental illness, the Executive shall have been
absent from his duties with the Company on a full-time basis for six months and
within 30 days after written notice of termination is thereafter given by the
Company the Executive shall not have returned to the full-time performance of
the Executive's duties, the Company may terminate this Agreement for
"Disability."

                  (c)      RETIREMENT. The term "Retirement" as used in this
Agreement shall mean termination by the Company or the Executive of the
Executive's employment based on the Executive's having reached age 65 or such
other age as shall have been fixed in any arrangement established with the
Executive's consent with respect to the Executive.

                  (d)      CAUSE. The Company may terminate the Executive's
employment for Cause. For purposes of this Agreement only, the Company shall
have "Cause" to terminate the Executive's employment hereunder only on the basis
of fraud, misappropriation or embezzlement on the part of the Executive.
Notwithstanding the foregoing, the Executive shall not be deemed to have been
terminated for Cause unless and until there shall have been delivered to the
Executive a copy of a resolution duly adopted by the affirmative vote of not
less than three-quarters of the membership of the Company's Board of Directors
(excluding the Executive if the Executive is then a member of the Board of
Directors) at a meeting of the Board called and held for the purpose (after
reasonable notice to the Executive and an opportunity for the Executive,
together with the Executive's counsel, to be heard before the Board), finding
that in the good faith opinion of the Board the Executive was guilty of conduct
set forth in the second sentence of this Section 3(d) and specifying the
particulars thereof in detail.

                  (e)      GOOD REASON. The Executive may terminate the
Executive's employment for Good Reason at any time during the term of this
Agreement. For purposes of this Agreement "Good Reason" shall mean any of the
following (without the Executive's express written consent):

                                       2

<PAGE>

                  (i)      the assignment to the Executive by the Company of
         duties inconsistent with the Executive's position, duties,
         responsibilities and status with the Company immediately prior to a
         Change in Control of the Company, or a change in the Executive's titles
         or offices as in effect immediately prior to a Change in Control of the
         Company, or any removal of the Executive from or any failure to reelect
         the Executive to any of such positions, except in connection with the
         termination of his employment for Disability, Retirement or Cause or as
         a result of the Executive's death or by the Executive other than for
         Good Reason;

                  (ii)     a reduction by the Company in the Executive's base
         salary as in effect on the date hereof or as the same may be increased
         from time to time during the term of this Agreement;

                  (iii)    a relocation of the Company's principal executive
         offices to a location outside of Nashville, Tennessee, or the
         Executive's relocation to any place other than the location at which
         the Executive performed the Executive's duties prior to a Change in
         Control of the Company, except for required travel by the Executive on
         the Company's business to an extent substantially consistent with the
         Executive's business travel obligations at the time of a Change in
         Control of the Company;

                  (iv)     any material breach by the Company of any provision
         of this Agreement;

                  (v)      any failure by the Company to obtain the assumption
         of this Agreement by any successor or assign of the Company; or

                  (vi)     any purported termination of the Executive's
         employment which is not effected pursuant to a Notice of Termination
         satisfying the requirements of Section 3(f), and for purposes of this
         Agreement, no such purported termination shall be effective.

                  (f)      NOTICE OF TERMINATION. Any termination by the Company
pursuant to Section 3(b), 3(c) or 3(d) shall be communicated by a Notice of
Termination. For purposes of this Agreement, a "Notice of Termination" shall
mean a written notice which indicates those specific termination provisions in
this Agreement relied upon and which sets forth in reasonable detail the facts
and circumstances claimed to provide a basis for termination of the Executive's
employment under the provisions so indicated. For purposes of this Agreement, no
such purported termination by the Company shall be effective without such Notice
of Termination.

                  (g)      DATE OF TERMINATION. "Date of Termination" shall mean
(a) if this Agreement is terminated by the Company for Disability, 30 days after
Notice of Termination is given to the Executive (provided that the Executive
shall not have returned to the performance of the Executive's duties on a
full-time basis during such 30-day period) or (b) if the Executive's employment
is terminated by the Company for any other reason, the date on which a Notice of
Termination is given.

         4.       SEVERANCE COMPENSATION UPON TERMINATION OF EMPLOYMENT. (a) If
the Company shall terminate the Executive's employment within eighteen months
following a

                                       3

<PAGE>

Change in Control other than pursuant to Section 3(b), 3(c) or 3(d) or if the
Executive shall terminate his employment within eighteen months following a
Change in Control for Good Reason, then the Company shall pay to the Executive
as severance pay in a lump sum, in cash, on the fifth day following the Date of
Termination, an amount equal to the sum of (i) 150% of the average of the
aggregate annual salary paid to the Executive by the Company during the three
calendar years preceding the Change in Control of the Company and (ii) 150% of
the highest bonus compensation paid to the Executive for any of the three
calendar years preceding the Change in Control of the Company; provided,
however, that if the lump sum severance payment under this Section 4, either
alone or together with other payments which the Executive has the right to
receive from the Company, would constitute a "parachute payment" (as defined in
Section 280G of the Internal Revenue Code of 1986, as amended (the "Code")),
such lump sum severance payment shall be reduced to the largest amount as will
result in no portion of the lump sum severance payment under this Section 4
being subject to the excise tax imposed by Section 4999 of the Code.

                  (b)      In addition to the lump sum payment provided in
Section 4(a), if the Company shall terminate the Executive's employment within
eighteen months following a Change in Control other than pursuant to Section
3(b), 3(c) or 3(d) or if the Executive shall terminate his employment within
eighteen months following a Change in Control for Good Reason, then the Company
shall provide to the Executive health insurance equivalent to that provided to
the Executive immediately prior to termination until the earlier of: (i)
eighteen months following the Date of Termination or (ii) such time as Executive
is employed by another employer and is covered or permitted to be covered by
benefit plans of another employer providing substantially similar coverage.

         5.       NO OBLIGATION TO MITIGATE DAMAGES; NO EFFECT ON OTHER
CONTRACTUAL RIGHTS. (a) The Executive shall not be required to mitigate damages
or the amount of any payment provided for under this Agreement by seeking other
employment or otherwise, nor shall the amount of any payment provided for under
this Agreement be reduced by any compensation earned by the Executive as the
result of employment by another employer after the Date of Termination, or
otherwise.

                  (b)      The provisions of this Agreement, and any payment
provided for hereunder, shall not reduce any amounts otherwise payable, or in
any way diminish the Executive's existing rights, or rights which would accrue
solely as a result of the passage of time, under any benefit plan, incentive
plan or stock option plan, employment agreement or other contract, plan or
arrangement.

         6.       SUCCESSOR TO THE COMPANY. (a) The Company will require any
successor or assign (whether direct or indirect, by purchase, merger,
consolidation or otherwise) to all or substantially all of the business and/or
assets of the Company, by agreement in form and substance satisfactory to the
Executive, expressly, absolutely and unconditionally to assume and agree to
perform this Agreement in the same manner and to the same extent that the
Company would be required to perform it if no such succession or assignment had
taken place. Any failure of the Company to obtain such agreement prior to the
effectiveness of any such succession or assignment shall be a material breach of
this Agreement and shall entitle the Executive to

                                       4

<PAGE>

terminate the Executive's employment for Good Reason. As used in this Agreement,
"Company" shall mean the Company as hereinbefore defined and any successor or
assign to its business and/or assets as aforesaid which executes and delivers
the agreement provided for in this Section 6 or which otherwise becomes bound by
all the terms and provisions of this Agreement by operation of law. If at any
time during the term of this Agreement the Executive is employed by any
corporation, a majority of the voting securities of which is then owned by the
Company, "Company" as used in Sections 3, 4, 11 and 12 hereof shall in addition
include such employer. In such event, the Company agrees that it shall pay or
shall cause such employer to pay any amounts owed to the Executive pursuant to
Section 4 hereof.

                  (b)      This Agreement shall inure to the benefit of and be
enforceable by the Executive's personal and legal representatives, executors,
administrators, successors, heirs, distributees, devisees and legatees. If the
Executive should die while any amounts are still payable to him hereunder, all
such amounts, unless otherwise provided herein, shall be paid in accordance with
the terms of this Agreement to the Executive's devisee, legatee, or other
designee or, if there be no such designee, to the Executive's estate.

         7.       NOTICE. For purposes of this Agreement, notices and all other
communications provided for in the Agreement shall be in writing and shall be
deemed to have been duly given when delivered or mailed by United States
registered mail, return receipt requested, postage prepaid, as follows:

                  If to the Company:

                           O'Charley's Inc.
                           3038 Sidco Drive
                           Nashville, Tennessee 37204
                           Attention: President

                  If to the Executive:

                           A. Chad Fitzhugh
                           3038 Sidco Drive
                           Nashville, Tennessee 37204

or such other address as either party may have furnished to the other in writing
in accordance herewith, except that notices of change of address shall be
effective only upon receipt.

         8.       MISCELLANEOUS. No provisions of this Agreement may be
modified, waived or discharged unless such waiver, modification or discharge is
agreed to in a writing signed by the Executive and the Company. No waiver by
either party hereto at any time of any breach by the other party hereto of, or
compliance with, any condition or provision of this Agreement to be performed by
such other party shall be deemed a waiver of similar or dissimilar provisions or
conditions at the same or at any prior or subsequent time. No agreements or
representations, oral or otherwise, express or implied, with respect to the
subject matter hereof have been made by

                                       5

<PAGE>

either party which are not set forth expressly in this Agreement. This Agreement
shall be governed by and construed in accordance with the laws of the State of
Tennessee.

         9.       VALIDITY. The invalidity or unenforceability of any provisions
of this Agreement shall not affect the validity or enforceability of any other
provision of this Agreement, which shall remain in full force and effect.

         10.      COUNTERPARTS. This Agreement may be executed in one or more
counterparts, each of which shall be deemed to be an original but all of which
together will constitute one and the same instrument.

         11.      LEGAL FEES AND EXPENSES. In the event either party hereto
shall institute litigation against the other party hereto relating to the
interpretation or enforcement of this Agreement, the prevailing party in such
litigation shall be entitled to recover from the other party any and all
attorneys' and related fees and expenses incurred by the prevailing party in
such litigation.

         12.      CONFIDENTIALITY. The Executive shall retain in confidence any
and all confidential information known to the Executive concerning the Company
and its business so long as such information is not otherwise publicly
disclosed. The provisions of this Section 12 are not intended to restrict the
ability of the Executive following termination of employment for any reason to
engage in any business which is, directly or indirectly, competitive with the
business conducted by the Company.

         IN WITNESS WHEREOF, the parties have executed this Agreement as of the
date first above written.

                                           O'CHARLEY'S INC.

                                           By: /s/ Gregory L. Burns
                                              ----------------------------------
                                           Name: Gregory L. Burns
                                                --------------------------------
                                           Title: Chief Executive Officer
                                                 -------------------------------

                                           /s/ A. Chad Fitzhugh
                                           -------------------------------------
                                           A. Chad Fitzhugh


                                       6