<PAGE>

                                                                    Exhibit 10.1

                     SEVERANCE AGREEMENT AND GENERAL RELEASE

      THIS SEVERANCE AGREEMENT AND GENERAL RELEASE ("Agreement") is made and
entered into this 24th day of June, 2004, by and between William E. Hall, Jr.
(hereinafter referred to as "Mr. Hall") and O'Charley's Inc. (hereinafter
referred to as "O'Charley's").

                                   WITNESSETH:

      WHEREAS, Mr. Hall currently serves as President, O'Charley's Concept for
O'Charley's and has expressed his desire to resign his employment, effective
June 27, 2004; and

      WHEREAS, O'Charley's has expressed its willingness to accept Mr. Hall's
resignation from employment upon the following terms and conditions; and

      WHEREAS, after a period of negotiations between them, the parties have
reached an agreement by which Mr. Hall will resign his employment effective June
27, 2004;

      NOW, THEREFORE, in consideration of the premises and mutual promises
herein contained, it is agreed as follows:

1.    The recitals set forth above are true and accurate, and by signing this
      Agreement, Mr. Hall hereby resigns his employment effective June 27, 2004
      (the "Effective Date").

2.    This Agreement is not and shall not be construed as an admission by
      O'Charley's of any fact or conclusion of law. Without limiting the general
      nature of the previous sentence, this Agreement shall not be construed as
      an admission that O'Charley's, its subsidiaries, related entities, or any
      of its or their officers, directors, managers, agents, or employees have
      violated any law or regulation or have violated any contract, express or
      implied.

3.    Mr. Hall represents and warrants that he has no knowledge of any practice
      engaged in by O'Charley's, its subsidiaries or related entities that is or
      was a violation in any material respect of any applicable state law or
      regulations or of any federal law or regulations. To the extent that Mr.
      Hall has knowledge of any such practices, Mr. Hall represents and warrants
      that Mr. Hall already has notified O'Charley's in writing of such alleged
      practices.

4.    Mr. Hall represents and warrants that he has not filed any complaint(s) or
      charge(s) against O'Charley's with the Equal Employment Opportunity
      Commission or the state commission empowered to investigate claims of
      employment discrimination, the United States Department of Labor, the
      Office of Federal Contract Compliance Programs, or with any other local,
      state or federal agency or court, and that if any such agency or court
      assumes jurisdiction of any complaint(s) or charge(s) against O'Charley's
      on behalf of Mr. Hall, Mr. Hall will request such agency or court to
      withdraw from the matter, and Mr.

<PAGE>

      Hall will refuse any benefits derived therefrom. This Agreement will not
      affect Mr. Hall's right to hereafter file a charge with or otherwise
      participate in an investigation or proceeding conducted by the Equal
      Employment Opportunity Commission regarding matters which arose after this
      date and which are not the subject of this Agreement.

5.    Mr. Hall represents and agrees that Mr. Hall is fully aware of his rights
      and is advised to discuss any and all aspects of this Agreement with his
      attorney, that Mr. Hall has consulted with his attorney regarding this
      Agreement, or has chosen voluntarily not to do so, that he has carefully
      read and fully understands all of the provisions of this Agreement, and
      that, in consideration of the provisions hereof, Mr. Hall agrees to enter
      into this Agreement. Mr. Hall represents and acknowledges that prior to
      the execution of this Agreement, he has been provided a period of
      twenty-one (21) days within which to consider the Agreement.

6.    Mr. Hall hereby irrevocably and unconditionally releases, acquits and
      forever discharges O'Charley's, its subsidiaries, and related entities,
      and each of their respective shareholders, successors, assigns, agents,
      directors, officers, employees, representatives, and attorneys, and all
      persons acting by, through, under or in concert with any of them
      (collectively, the "Released Parties"), or any of them, from any and all
      charges, complaints, claims, liabilities, obligations, promises,
      agreements, controversies, damages, actions, causes of action, suits,
      rights, demands, costs, losses, debts and expenses (including attorney's
      fees and costs actually incurred), of any nature whatsoever, known or
      unknown ("Claims"), which Mr. Hall now has, owns, holds, or claims to
      have, own, or hold, or which Mr. Hall at any time heretofore had, owned,
      or held, or claimed to have, own, or hold. Such Claims include those under
      local, state or federal law, Executive Order, or at common law including,
      but not limited to, the Age Discrimination in Employment Act (ADEA). This
      provision does not include the release of future charges before the Equal
      Employment Opportunity Commission regarding matters which arose after this
      date and which are not the subject of this Agreement. This provision
      further does not include the release of Claims with respect to any vested
      benefits under a plan governed by the Employee Retirement Income Security
      Act ("ERISA") or any Claim related to the rights and benefits granted by
      the express terms of this Agreement.

7.    O'Charley's hereby irrevocably and unconditionally releases, acquits and
      forever discharges Mr. Hall from any and all charges, complaints, claims,
      liabilities, obligations, promises, agreements, controversies, damages,
      actions, causes of action, suits, rights, demands, costs, losses, debts
      and expenses (including attorney's fees and costs actually incurred), of
      any nature whatsoever, known or unknown ("Claims"), which O'Charley's now
      has, owns, holds, or claims to have, own, or hold, or which O'Charley's at
      any time heretofore had, owned, or held, or claimed to have, own, or hold;
      provided, the provisions of this paragraph 7 shall not apply to any Claim
      (i) arising out of the commission by Mr. Hall of any act of fraud,
      embezzlement or material dishonesty, (ii) any Claim related to the Note
      (as defined in paragraph 15), or (iii) any Claim related to any breach of
      the representations, warranties or covenants of Mr. Hall provided for by
      the express terms of this Agreement.

                                       2
<PAGE>

8.    Mr. Hall agrees that, as of the date of the signing of this Agreement, Mr.
      Hall will not, without O'Charley's prior written consent, directly or
      indirectly, solicit to hire or hire (or cause to leave the employ of
      O'Charley's) (i) for a period of twelve (12) months from the Effective
      Date, any salaried employee of O'Charley's or its subsidiaries and (ii)
      for a period of twenty-four (24) months from the Effective Date, any
      general manager or regional supervisor of any O'Charley's or Stoney River
      Legendary Steaks restaurant. The agreements set forth in this paragraph 8,
      together with those contained in paragraph 9, are sometimes hereinafter
      collectively referred to as the "Restrictive Agreements."

9.    Mr. Hall also recognizes that, as President, O'Charley's Concept he has
      had access to, was provided in detail with, and used throughout his
      employment with O'Charley's, certain confidential and proprietary business
      information. Such information includes but is not limited to business
      strategy, pricing information, branding strategy, site location, vendor
      information, market analysis and evaluation, and other such proprietary
      and confidential business information (hereinafter "Confidential and
      Proprietary Business Information"). Mr. Hall agrees that, without
      O'Charley's prior written consent, he will not divulge, disclose, publish
      or disseminate in any manner, directly or indirectly, any such
      Confidential and Proprietary Business Information to any other person or
      entity.

10.   Mr. Hall agrees that the Restrictive Agreements are reasonable and
      supported by adequate consideration, which but for his agreement herein,
      Mr. Hall would not be entitled to receive. Mr. Hall further agrees that
      the Restrictive Agreements are necessary for the protection of
      O'Charley's, its business and its employees, and properly balance his
      personal interest in future employment and the various interests related
      to O'Charley's, its business and its employees. Mr. Hall further agrees
      that if he breaches any of the Restrictive Agreements, such breach likely
      will not have an adequate remedy at law and that O'Charley's shall be
      entitled, in addition to all other legal remedies available to it, to
      apply to and obtain from a court of competent jurisdiction an injunction
      against any violation thereof and, if successful, to recover all costs of
      such action, including a reasonable attorney's fee. These rights and
      remedies of O'Charley's shall be cumulative and not alternative.

11.   Mr. Hall agrees to indemnify and hold each and all of the Released Parties
      harmless from and against any and all loss, costs, damage, or expense,
      including, without limitation, attorneys fees, incurred by the Released
      Parties, or any of them, arising out of Mr. Hall's breach of the
      representations, warranties and covenants made by Mr. Hall in paragraphs
      1, 3, 4, 5, 6, 8, 9, 10, 12, 14, 16, 17 and 18 of this Agreement or the
      fact that any representation made by him herein was false when made.

12.   Given the nature of his position with O'Charley's, Mr. Hall likely has
      been privy to information and events during his tenure with O'Charley's
      that may not be documented. For example, Mr. Hall is aware that
      O'Charley's currently is the subject of litigation relating to certain
      customers and employees at one of its O'Charley's restaurants in
      Knoxville, Tennessee being exposed to the Hepatitis A virus and may be the
      subject of

                                       3
<PAGE>

      additional litigation relating to this incident (the "Hepatitis A
      Litigation"). Mr. Hall agrees to provide such information and assistance
      as O'Charley's and its legal counsel shall reasonably request regarding
      matters related to his responsibilities while employed by O'Charley's and
      to assist in O'Charley's defense of the Hepatitis A Litigation, it being
      understood that such assistance shall not unreasonably interfere with Mr.
      Hall's other business responsibilities following the date hereof.
      O'Charley's agrees to indemnify and advance expenses to Mr. Hall in the
      event he is made a party to a proceeding related to the Hepatitis A
      Litigation from and after the Effective Date to the same extent as if Mr.
      Hall continued to be an employee of O'Charley's.

13.   O'Charley's will provide to Mr. Hall the following:

      (a)   Fourteen months of salary continuation, including car and gas
            allowance at the existing rate, at the rate of $4,614.62 per week,
            to be paid weekly, for the period of June 28, 2004 to August 28,
            2005, and from which O'Charley's will make applicable legal
            withholdings;

      (b)   If Mr. Hall elects to continue health insurance coverage pursuant to
            his "COBRA" right after the Effective Date, O'Charley's will pay the
            premiums for such coverage during the twelve month period following
            the Effective Date ending June 26, 2005.

      (c)   O'Charley's will reimburse Mr. Hall for the fees of legal counsel
            engaged by Mr. Hall to review the terms of this Agreement, not to
            exceed $3,000.

14.   Attached as Schedule A is a listing of all stock options and restricted
      stock awards held by Mr. Hall as of the Effective Date. Mr. Hall agrees
      that Schedule A accurately reflects all equity awards and their respective
      terms held by him as of the date hereof and as amended hereby. All such
      awards were granted pursuant to the terms of the O'Charley's 2000 Stock
      Incentive or O'Charley's 1990 Employee Stock Plan. Pursuant to such plans,
      upon the Effective Date, each of the stock options and restricted stock
      awards would automatically terminate; however, notwithstanding the
      provisions of such plans, O'Charley's agrees that 3,462 of the 9,890
      shares subject to the Restricted Stock Award dated February 19, 2003 shall
      vest as of the Effective Date and to extend until June 30, 2005 the time
      within which Mr. Hall shall have the right to exercise any and all stock
      options listed on Schedule A to the extent they are exercisable as of the
      Effective Date (it being understood that all or any portion of any stock
      options, restricted stock awards or other equity based awards which are
      not exercisable or vested as of the Effective Date shall terminate as of
      such date in accordance with their terms).

15.   Mr. Hall is a party to a Secured Promissory Note dated February 13, 2002
      (the "Note") pursuant to which he is obligated to pay O'Charley's the
      principal amount of $74,000 plus accrued interest. The Note shall remain
      in full force and effect notwithstanding the termination of Mr. Hall's
      employment and the provisions of this Agreement and, accordingly, Mr. Hall
      agrees to pay the entire outstanding principal balance, together with

                                       4
<PAGE>

      all accrued and unpaid interest, under the Note on February 13, 2005 in
      accordance with the terms of the Note.

16.   Mr. Hall represents that he has not heretofore assigned or transferred, or
      purported to assign or transfer to any person or entity, any claim or any
      portion thereof or interest therein.

17.   Mr. Hall represents and acknowledges that in executing this Agreement he
      does not rely and has not relied upon any other representation or
      statement made by any of the Released Parties or by any of the Released
      Parties' agents, representatives or attorneys, except as set forth herein,
      with regard to the subject matter, basis or effect of this Agreement.

18.   Mr. Hall agrees to maintain absolute confidentiality and secrecy
      concerning the terms of this Agreement and the circumstances leading to
      this Agreement, and he will not reveal, or disseminate by publication in
      any manner whatsoever, this document or any matter pertaining to it to any
      other person, including, but not limited to, any past or present employees
      of O'Charley's or any media representative except as required by legal
      process. This confidentiality provision does not apply to communications
      necessary between legal and financial planners or tax preparers or spouse
      so long as Mr. Hall requires those persons to accept the terms of this
      confidentiality agreement before being provided the information contained
      herein.

19.   This Agreement shall be binding upon O'Charley's, Mr. Hall and upon Mr.
      Hall's heirs, administrators, representatives, executors, successors, and
      assigns, and shall inure to the benefit of the Released Parties and each
      of them, and to their heirs, administrators, representatives, executor,
      successors and assigns.

20.   This Agreement shall in all respects be interpreted, enforced and governed
      under the laws of the State of Tennessee.

21.   Mr. Hall shall have seven (7) days following the execution of this
      Agreement during which to revoke the Agreement. This Agreement shall
      become effective and irrevocable only after the seven (7) day period has
      expired and only absent a timely and effective revocation.

22.   Should any provision of this Agreement be declared or be determined by any
      court to be illegal or invalid, the validity of the remaining parts,
      terms, or provisions shall not be affected thereby and said illegal or
      invalid part, term or provision shall be deemed not to be a part of this
      Agreement.

23.   Should litigation arise in relation to this Agreement, then the prevailing
      party shall recover from the other the entire amount of fees, expenses and
      costs, including, but not limited to, attorney fees, depositions, travel,
      expert costs, and the like.

                                       5
<PAGE>

24.   This Agreement sets forth the entire agreement between the parties hereto.

                                            O'CHARLEY'S INC.

                                            By: /s/ Gregory L. Burns
                                                --------------------------------

                                            WILLIAM E. HALL, JR.

                                                /s/ William E. Hall, Jr.
                                                --------------------------------
                                            Date: 6/24/04

                                       6
<PAGE>

                                   SCHEDULE A

                              LIST OF EQUITY AWARDS

<TABLE>
<CAPTION>
                                                                                                  NUMBER OF
                                                                                                    SHARES
                                      GRANT               EXERCISE        SHARES SUBJECT        EXERCISABLE ON
      TYPE OF AWARD                   DATE                 PRICE             TO AWARD           EFFECTIVE DATE
--------------------------          ----------            --------        --------------        --------------
<S>                                 <C>                   <C>             <C>                   <C>
Non-Qualified Stock Option          11/12/1997             $11.33              45,000               42,863

Non-Qualified Stock Option           2/17/1999             $15.25              50,000               35,125

Restricted Stock Award               2/13/2002                N/A               7,085                2,363

Non-Qualified Stock Option           2/19/2003             $21.19              19,780                4,945

Restricted Stock Award               2/19/2003                N/A               9,890                3,462
</TABLE>

                                       7