<PAGE>   1
                                                                   EXHIBIT 10.20

                               SEVERANCE AGREEMENT


          THIS SEVERANCE AGREEMENT (this "AGREEMENT") is made and entered into
as of June 21, 1999 by and between Pacific Sunwear of California, Inc., a
California corporation (the "COMPANY"), and Mark A. Hoffman ("EXECUTIVE").

                                   WITNESSETH:

          WHEREAS, the Executive is employed by the Company in the capacity of
Chief Operating Officer, and as such the Board of Directors of the Company (the
"BOARD") deems it in the best interest of the Company to offer this Agreement to
the Executive; and

          WHEREAS, the Company and the Executive wish to provide for the
continuation of certain payments and benefits to Executive upon the termination
of Executive's employment under specified circumstances, and would like to set
forth the terms relating to a release by Executive of any claims Executive may
have against the Company;

          NOW, THEREFORE, in consideration of the mutual promises and covenants
herein contained, and good and valuable other consideration the receipt and
sufficiency of which are hereby acknowledged, the parties hereto hereby agree as
follows:

          1. POST-TERMINATION BENEFITS.

          a. Salary; Bonus. If the employment of Executive by the Company is
terminated without Cause (as defined below), Executive shall continue to receive
from the Company payment of Executive's base salary for a period of nine months
following the date of termination of Executive's employment (the "TERMINATION
DATE"). Such payments of base salary shall be payable to Executive semi-monthly
in arrears. For purposes of this Agreement, "Cause" shall mean only that (i)
Executive has refused to perform or discharge his material obligations or duties
hereunder for 30 days after notice from the Board of such refusal, or (ii)
Executive has engaged in illegal or other wrongful conduct substantially
detrimental to the business or reputation of the Company. In addition to the
foregoing, if the employment of Executive is terminated by the Company without
Cause at any time during a fourth fiscal quarter of the Company, the Company
shall also pay to Executive in a single payment within 60 days of the end of the
Company's fiscal year a "Pro Rata Portion of the Bonus." "Pro Rata Portion of
the Bonus" means an amount equal to any bonus to which Executive would have been
entitled had Executive remained an employee for the balance of the fiscal year
in which his employment terminated multiplied by a fraction, the numerator of
which is the number of days from February 1 to the date of Executive's
termination, and the denominator of which is 365.

          It shall be a condition to the obligations of the Company to make the
payments and provide the other benefits required hereunder, that Executive
execute and deliver to the Company an Unconditional Release Agreement with the
Company in substantially the form attached as Exhibit A (the "RELEASE
AGREEMENT") and that, thereafter, no revocation of the release of age
discrimination claims be made by Executive.



<PAGE>   2

          b. Medical Benefits. If the employment of Executive by the Company is
terminated without Cause, the Company shall make available to Executive such
medical insurance coverage as may be provided to Executive by the Company
immediately prior to the termination of Executive's employment with the Company
(or such Company insurance coverage which is consistent with the coverage in
place from time to time for comparable executives of the Company). The Company
shall provide the medical insurance coverage to Executive for a period of nine
(9) months following the Termination Date. Upon Executive's attainment of new
employment (in any capacity) and qualification for medical insurance coverage
pursuant to such new employment, the Company shall no longer be obligated to
provide medical insurance coverage of any type or nature whether or not a period
of nine months has lapsed since Executive's termination. Executive agrees to
immediately notify the Company concerning his attainment of new employment and
medical insurance coverage.

          c. Payment for Medical Benefits. The Company shall pay all costs and
expenses associated with providing the medical insurance; provided, however,
that Executive shall be obligated to pay to the Company monthly an amount equal
to the aggregate amount of all co-payments and/or fees relating to insurance
coverage which Executive was responsible for prior to the termination of his
employment, whether such co-payments and fees were paid to the Company or
directly to an insurance provider.

          2. AT-WILL. It is expressly understood and acknowledged by Executive
that Executive's employment by the Company is "at-will" and nothing in this
Agreement alters the "at-will" nature of Executive's employment. Executive
acknowledges that the Company may terminate his employment at any time with or
without Cause; provided, however that if the termination is without Cause,
Executive will be entitled to the benefits described herein.

          3. COUNTERPARTS. This Agreement may be executed in one or more
counterparts. All of such counterparts shall constitute one and the same
agreement and shall become effective when a copy signed by each party has been
delivered to the other party.

          4. MISCELLANEOUS. This Agreement constitutes the entire agreement of
the parties hereto relating to the subject matter hereof, and there are no
written or oral terms or representations made by either party other than those
contained herein. This Agreement supersedes all prior agreements between the
parties concerning the subject matter hereof, including that certain Severance
Agreement dated February 6, 1996, between the Company and Executive which is
hereby deemed terminated as of the date of this Agreement. This Agreement may
only be amended in writing signed by both parties. No waiver by any party of any
breach of this Agreement shall be deemed to be a waiver by any party of any
preceding or succeeding breach. The validity, interpretation, performance and
enforcement of this Agreement shall be governed by the laws of the State of
California without regard to conflicts of law principles. The headings contained
herein are for reference purposes only and shall not in any way affect the
meaning or interpretation of this Agreement.

                                        2

<PAGE>   3


          IN WITNESS WHEREOF, the parties have executed this Amendment
as of the date first above written.

                                           "COMPANY"

                                           PACIFIC SUNWEAR OF CALIFORNIA, INC.,
                                           a California corporation



                                           By:    /s/ GREG WEAVER
                                                  ------------------------------

                                           Name:  Greg Weaver
                                                  ------------------------------

                                           Title: Chairman of the Board
                                                  ------------------------------


                                           "EXECUTIVE"


                                           /s/ MARK A. HOFFMAN
                                           -------------------------------------
                                           Mark A. Hoffman


                                        3