<PAGE>   1
                       SECURITIES AND EXCHANGE COMMISSION
                              WASHINGTON D.C 20549

                                    FORM 10-Q

[X]   QUARTERLY REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES
      EXCHANGE ACT OF 1934

                   FOR THE QUARTERLY PERIOD ENDED MAY 4, 1997

                                       OR

[ ]   TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES
      EXCHANGE ACT OF 1934


                         Commission file number 0-21296

                      PACIFIC SUNWEAR OF CALIFORNIA, INC.


                CALIFORNIA                                       95-3759463
        (State of Incorporation)                              (I.R.S Employer 
                                                             Identification No.)

         5037 EAST HUNTER AVENUE
           ANAHEIM, CALIFORNIA                                      92807
  (Address of principal executive offices)                        (Zip code)



                                 (714) 693-8066
              (Registrant's telephone number, including area code)

Indicate by check mark whether the registrant (1) has filed all reports required
to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during
the preceding 12 months (or for such shorter period that the registrant was
required to file such reports); and (2) has been subject to such filing
requirements for the past 90 days.   YES  X   NO
                                         ---     ---

         The number of shares outstanding of the registrant's Common Stock, par
value $.01 per share, at May 23, 1997 was 8,155,140.


<PAGE>   2
                       PACIFIC SUNWEAR OF CALIFORNIA, INC.

                                    FORM 10-Q

                        FOR THE QUARTER ENDED MAY 4, 1997

                                      INDEX

<TABLE>
<CAPTION>
                                                                                                      Page
                                                                                                      ----
<S>               <C>                                                                                 <C>
PART I.           FINANCIAL INFORMATION

Item 1.           Financial Statements:
                    Balance Sheets as of May 4, 1997 (unaudited) and February 2, 1997................    3
                    Statements of Operations (unaudited) for the thirteen weeks
                      ended May 4, 1997 and May 5, 1996..............................................    4
                    Statements of Cash Flows (unaudited) for the thirteen weeks
                      ended May 4, 1997 and May 5, 1996..............................................    5
                    Notes to Financial Statements....................................................    6

Item 2            Management's Discussion and Analysis of Financial Condition
                    and Results of Operations........................................................  7-9

PART II.          OTHER INFORMATION

Item 1.           Legal Proceedings..................................................................   10
Item 2.           Changes in Securities..............................................................   10
Item 3.           Defaults Upon Senior Securities....................................................   10
Item 4.           Submission of Matters to a Vote of Security Holders................................   10
Item 5.           Other Information..................................................................   10
Item 6.           Exhibits and Reports on Form 8-K...................................................   10

                  SIGNATURE PAGE.....................................................................   11
</TABLE>

                                        2

<PAGE>   3
                       PACIFIC SUNWEAR OF CALIFORNIA, INC.

                                 BALANCE SHEETS

                                     ASSETS
<TABLE>
<CAPTION>
                                                                                   MAY 4,          FEBRUARY 2,
                                                                                    1997              1997
                                                                                 ------------      ------------
                                                                                 (unaudited)
<S>                                                                              <C>               <C>
CURRENT ASSETS:
  Cash and cash equivalents                                                      $  7,196,956      $  9,962,626
  Accounts receivable                                                                 803,367           583,811
  Prepaid income taxes                                                                304,540                --
  Merchandise inventories                                                          22,461,924        19,760,412
  Prepaid expenses, includes $1,948,487 and $1,910,681 of prepaid rent,
     respectively                                                                   3,036,047         3,216,160
  Deferred taxes                                                                    1,358,733         1,358,733
                                                                                 ------------      ------------
     Total current assets                                                          35,161,567        34,881,742

PROPERTY AND EQUIPMENT:
  Leasehold improvements                                                           27,210,834        25,210,439
  Furniture, fixtures and equipment                                                21,901,491        20,244,954
                                                                                 ------------      ------------
                                                                                   49,112,325        45,455,393
  Less accumulated depreciation and amortization                                  (17,419,688)      (15,952,174)
                                                                                 ------------      ------------
     Net property and equipment                                                    31,692,637        29,503,219

OTHER ASSETS:
  Goodwill, net of accumulated amortization of $298,886 and $292,165,
      respectively                                                                    789,857           796,578

  Deposits and other assets                                                           694,358           523,018
                                                                                 ------------      ------------
     Total other assets                                                             1,484,215         1,319,596
                                                                                 ------------      ------------
       Total assets                                                              $ 68,338,419      $ 65,704,557
                                                                                 ============      ============

                                     LIABILITIES AND SHAREHOLDERS' EQUITY

CURRENT LIABILITIES:
  Accounts payable                                                               $  8,502,095      $  6,686,561
  Accrued liabilities (Note 4)                                                      5,245,289         6,035,689

  Income taxes payable                                                                     --           469,258
                                                                                 ------------      ------------
     Total current liabilities                                                     13,747,384        13,191,508
DEFERRED COMPENSATION                                                                 654,997           371,057
DEFERRED RENT                                                                       3,254,754         3,139,487
DEFERRED TAXES                                                                      1,456,463         1,456,463

SHAREHOLDERS' EQUITY:
  Preferred Stock, par value $.01; authorized, 5,000,000 shares; none issued
     and outstanding
  Common Stock, par value $.01; authorized, 22,500,000 shares; issued and
     outstanding, 8,135,134  and 8,092,107 shares, respectively                        81,352            80,921
  Additional paid-in capital                                                       31,291,852        30,737,782
  Retained earnings                                                                17,851,617        16,727,339
                                                                                 ------------      ------------
     Total shareholders' equity                                                    49,224,821        47,546,042
                                                                                 ------------      ------------
       Total liabilities and shareholders' equity                                $ 68,338,419      $ 65,704,557
                                                                                 ============      ============
</TABLE>


                            See accompanying notes


                                        3

<PAGE>   4
                          PACIFIC SUNWEAR OF CALIFORNIA, INC

                               STATEMENTS OF OPERATIONS
                                      (UNAUDITED)
<TABLE>
<CAPTION>
                                                     THIRTEEN WEEKS ENDED
                                                 -----------------------------
                                                  MAY 4, 1997      MAY 5, 1996
                                                 ------------     ------------
<S>                                              <C>              <C>         
Net sales                                        $ 38,932,736     $ 27,640,975

Cost of goods sold, including buying,
  distribution, and occupancy costs                27,225,628       20,362,870
                                                 ------------     ------------
Gross margin                                       11,707,108        7,278,105

Selling, general and administrative expenses        9,942,532        7,576,340
                                                 ------------     ------------
Operating income (loss)                             1,764,576         (298,235)
Interest income                                        95,702           29,659
                                                 ------------     ------------
Income (loss) before income tax expense          
  (benefit)                                         1,860,278         (268,576)

Income tax expense(benefit)                           736,000          (99,000)
                                                 ------------     ------------
Net income (loss)                                $  1,124,278     $   (169,576)
                                                 ============     ============
Net income (loss) per common and
    common equivalent share                      $       0.13     $      (0.02)
                                                 ------------     ------------
Weighted average common and common
    equivalent shares outstanding                   8,442,791        7,892,310
                                                 ============     ============
</TABLE>


                                See accompanying notes


                                        4

<PAGE>   5
                       PACIFIC SUNWEAR OF CALIFORNIA, INC

                            STATEMENTS OF CASH FLOWS
                                   (UNAUDITED)
<TABLE>
<CAPTION>
                                                                          THIRTEEN WEEKS ENDED
                                                                      ----------------------------
                                                                      MAY 4, 1997      MAY 5, 1996
                                                                      -----------      -----------
<S>                                                                   <C>              <C>
CASH FLOWS FROM OPERATING ACTIVITIES:
Net income(loss)                                                      $ 1,124,278      $  (169,576)
Adjustments to reconcile net income (loss) to net cash
  provided by operating activities:

   Depreciation and amortization                                        1,474,235        1,234,916

   Change in:
     Accounts receivable                                                 (219,556)        (316,597)
     Merchandise inventories                                           (2,701,512)      (2,707,960)
     Prepaid expenses                                                     180,113           57,426
     Deposits and other assets                                           (171,340)         (29,919)
     Accounts payable                                                   1,815,534        4,264,612
     Accrued liabilities                                                 (790,400)         642,704
     Income taxes and deferred income taxes                              (324,350)        (557,150)
     Deferred rent                                                        115,267          131,760
     Deferred compensation                                                283,940           56,786
                                                                      -----------      -----------
        Net cash provided by operating activities                         786,209        2,607,002

CASH FLOWS FROM INVESTING ACTIVITIES:

     Investment in property and equipment                              (3,656,932)      (1,521,251)
                                                                      -----------      -----------
        Net cash used in investing activities                          (3,656,932)      (1,521,251)

CASH FLOWS FROM FINANCING ACTIVITIES:

     Principal payments under loan agreement                                   --         (781,250)
     Proceeds from exercise of stock options                              105,053          488,276
                                                                      -----------      -----------
        Net cash provided by (used in) financing activities               105,053         (292,974)
                                                                      -----------      -----------
NET INCREASE(DECREASE) IN CASH AND CASH EQUIVALENTS                    (2,765,670)         792,777

  CASH AND CASH EQUIVALENTS, beginning of period                        9,962,626        4,315,842
                                                                      -----------      -----------
  CASH AND CASH EQUIVALENTS, end of period                            $ 7,196,956      $ 5,108,619
                                                                      ===========      ===========

SUPPLEMENTAL DISCLOSURES OF CASH FLOW INFORMATION:

Cash paid during the period for:

  Interest                                                            $       478      $    10,685
  Income taxes                                                        $ 1,060,350      $   458,150
--------------------------------------------------------------------------------------------------
</TABLE>

Non-cash transaction: During the thirteen weeks ended May 4, 1997, the Company
recorded an increase to additional paid-in capital of $449,448 related to tax
benefits associated with the exercise of non-qualified stock options.


                             See accompanying notes


                                        5

<PAGE>   6
                       PACIFIC SUNWEAR OF CALIFORNIA, INC.

                          NOTES TO FINANCIAL STATEMENTS

NOTE 1 - BASIS OF PRESENTATION

         The accompanying financial statements are unaudited except for the
February 2, 1997 balance sheet. These statements have been prepared in
accordance with generally accepted accounting principles for interim financial
information and with the instructions to Form 10-Q and Rule 10-01 of Regulation
S-X. Accordingly, they do not include all of the information and footnotes
required by generally accepted accounting principles for complete financial
statements.

         The Company's fiscal year is the 52- or 53-week period which ends on
the Sunday closest to the end of January. "Fiscal 1997" is a 52-week period
which ends February 1, 1998.

         In the opinion of management, all adjustments consisting only of normal
recurring entries necessary for a fair presentation have been included. The
preparation of financial statements in conformity with generally accepted
accounting principles necessarily requires management to make estimates and
assumptions that affect the reported amounts of assets and liabilities and
disclosure of contingent assets and liabilities at the date of the financial
statements and reported expenses during the reported period. Actual results
could differ from these estimates. The results of operations for the thirteen
weeks ended May 4, 1997 are not necessarily indicative of the results that may
be expected for the fiscal year ending February 1, 1998. For further
information, refer to the financial statements and notes thereto as of and for
the years ended February 2, 1997, February 4, 1996 and January 29, 1995.

NOTE 2 - NET INCOME (LOSS) PER COMMON AND COMMON EQUIVALENT SHARE

         Net income (loss) per common and common equivalent share is based on
the weighted average number of common and common equivalent shares outstanding
during the relevant periods. For the thirteen weeks ended May 5, 1996, no effect
has been given to options outstanding under the Company's Stock Option Plan as
they were not dilutive.

         Stock Split - On October 9, 1996, the Company effected a three-for-two
stock split. Earnings per share and share outstanding amounts have been given
retroactive effect in these financial statements.

         In February 1997, the Financial Accounting Standards Board issued
Statement of Financial Accounting Standards No. 128, "Earnings Per Share" ("SFAS
No. 128") which is effective for financial statements issued for periods ending
after December 15, 1997. SFAS No. 128 requires the disclosure of basic and
diluted earnings per share. For the thirteen weeks ended May 4, 1997, the amount
reported as net income per common and common equivalent share is not materially
different from that which would have been reported for basic and diluted
earnings per share in accordance with SFAS No. 128.

NOTE 3 - FEDERAL AND STATE INCOME TAX EXPENSE (BENEFIT)

         The combined federal and state income tax expense (benefit) were
calculated using estimated effective annual tax rates.

NOTE 4 - ACCRUED LIABILITIES

         Accrued liabilities consist of the following:

<TABLE>
<CAPTION>
                                                          MAY 4,      FEBRUARY 2,
                                                          1997           1997
                                                       -----------    ----------
<S>                                                    <C>            <C>       
Accrued compensation and benefits                      $1,932,561     $2,939,217
Reserve for expansion/relocation and closing costs      1,669,156      1,424,315
Other accrued liabilities                               1,643,572      1,672,157
                                                       ----------     ----------
                                                       $5,245,289     $6,035,689
                                                       ==========     ==========
</TABLE>


                                       6

<PAGE>   7
                MANAGEMENT'S DISCUSSION AND ANALYSIS OF FINANCIAL
                       CONDITION AND RESULTS OF OPERATIONS

RESULTS OF OPERATIONS

The thirteen weeks ended May 4, 1997 as compared to the thirteen weeks ended 
May 5, 1996

         Net Sales

         Net sales increased to $38.9 million for the first thirteen weeks of
fiscal 1997 from $27.6 million for the first thirteen weeks of fiscal 1996, an
increase of $11.3 million, or 40.9%. Of this $11.3 million increase, $5.8
million was attributable to net sales generated by new stores opened in fiscal
1996 and not yet included in the comparable store base, $4.5 million was
attributable to a 17.5% increase in comparable store net sales in the first
thirteen weeks of fiscal 1997, $.7 million was attributable to nine stores that
have been expanded or relocated to the larger format and not yet included in the
comparable store base, and $.6 million was attributable to net sales generated
by 11 new stores opened in fiscal 1997 and not yet included in the comparable
store base. Offsetting these increases was a $.3 million decrease attributable
to the closing of four stores. The increase in comparable store net sales was
primarily attributable to the addition of footwear and juniors to certain of the
Company's stores and, to a lesser extent, to increases in sales of young men's
merchandise. Net sales of footwear and juniors represented approximately 19% of
total net sales for the first thirteen weeks of fiscal 1997 compared to 6% for
the first thirteen weeks of fiscal 1996. The average retail price per unit sold
increased approximately 11% in the first thirteen weeks of fiscal 1997 compared
to the first thirteen weeks of fiscal 1996, primarily attributable to a change
in the mix of products sold, including the addition of footwear, an increase in
sales of pants as a percentage of net sales and a decrease in T-shirt sales as a
percentage of net sales.

         Gross Margin

         Gross margin, after buying, distribution and occupancy costs, increased
to $11.7 million for the first thirteen weeks of fiscal 1997 from $7.3 million
for the first thirteen weeks of fiscal 1996, an increase of $4.4 million, or
60.3%. As a percentage of net sales, gross margin increased to 30.1% from 26.4%.
Of this 3.7% increase, 2.5% was due to a decrease in occupancy costs as a
percentage of net sales which was related primarily to higher comparable store
net sales. In addition, net merchandise margins increased 1.0% as a percentage
of net sales for the first thirteen weeks of fiscal 1997 compared to the first
thirteen weeks of fiscal 1996 due primarily to an increase in initial markup and
a decrease in markdowns as a percentage of net sales. Furthermore, buying and
distribution costs decreased by .2% as a percentage of net sales.

         Selling, General and Administrative Expenses

         Selling, general and administrative expenses increased to $9.9 million
for the first thirteen weeks of fiscal 1997 from $7.6 million for the first
thirteen weeks of fiscal 1996, an increase of $2.3 million, or 30.3%. As a
percentage of net sales, these expenses decreased to 25.5% from 27.5%. Of this
2.0% decrease as a percentage of net sales, 1.9% was attributable to a decrease
in store selling expenses as a percentage of net sales primarily as a result of
an increase in comparable store net sales and higher total net sales and .6% was
due to a decrease in general and administrative expenses as a percentage of net
sales as a result of leveraging these expenses over higher total net sales.
Partially offsetting this decrease was an increase of .5% due to increased
expansion and relocation expenses and closing expenses as a percentage 
of net sales compared to the first thirteen weeks of fiscal 1996.

         Income Tax Expense (Benefit)

         Income tax expense was $.7 million in the first thirteen weeks of
fiscal 1997 compared to an income tax benefit of $.1 million for the first
thirteen weeks of fiscal 1996. The effective income tax rate for the first
thirteen weeks of fiscal 1997 was 39.6 % compared to a benefit of 36.9% for
the first thirteen weeks of fiscal 1996.


                                        7

<PAGE>   8
LIQUIDITY AND CAPITAL RESOURCES

         The Company has financed its operations from internally generated cash
flow, short-term borrowings and equity financing. The Company's primary capital
requirements have been for the construction of new stores, remodeling,
expansion, or relocation of selected stores and financing of inventories.
         
         Net cash provided by operating activities for the first thirteen weeks
of fiscal 1997 was $.8 million compared to $2.6 million for the first thirteen
weeks of fiscal 1996. The $1.8 million decrease was primarily attributable to a
decrease in accounts payable of $2.4 million and a decrease in accrued
liabilities of $1.4 million, offset by a $1.3 million increase in net income and
net increases in other items of $.7 million. Working capital at May 4, 1997 was
$21.4 million compared to $21.7 million at February 2, 1997, a decrease of $.3
million. Inventories at May 4, 1997 were $22.5 million compared to $19.8 million
at February 2, 1997, an increase of $2.7 million. This increase was primarily
related to opening 11 new stores, relocating two stores with 50% larger
average square footage than existing stores and the addition of juniors to
certain of the Company's existing stores. The increase in accounts payable of
$1.8 million at May 4, 1997 compared to February 2, 1997 was primarily
attributable to the increase in inventories at May 4, 1997.

         Net cash used in investing activities was $3.7 million for the first
thirteen weeks of fiscal 1997 compared to $1.5 million for the first thirteen
weeks of fiscal 1996. This increase was primarily due to net cash invested in
property and equipment for the first thirteen weeks of fiscal 1997 of $3.7
million compared to $1.5 million for the first thirteen weeks of fiscal 1996,
due to an increase in the number of new stores opened and, to a lesser extent,
an increase in the number of stores expanded/relocated in the first thirteen
weeks of fiscal 1997 compared to the first thirteen weeks of fiscal 1996.

         Net cash provided by (used in) financing activities for the first
thirteen weeks of fiscal 1997 was $.1 million compared to $(.3) million for the
first thirteen weeks of fiscal 1996. In the first thirteen weeks of fiscal 1997,
the Company made no borrowings or repayments under its loan agreement, compared
to the repayment of a term loan of $.8 million in the first thirteen weeks of
fiscal 1996. In the first thirteen weeks of fiscal 1997, the Company received
proceeds of $.1 million from the exercise of stock options compared to $.5
million in the first thirteen weeks of fiscal 1996. At May 4, 1997, the Company
had $3.1 million in letters of credit outstanding.

         The Company plans to open approximately 39 stores, and expand or
relocate approximately 13 existing stores during the remainder of fiscal 1997.
The Company estimates that capital expenditures during the remainder of fiscal
1997 will be approximately $12.0 million.

         The Company reviews the operating performance of its stores on an
ongoing basis to determine which stores, if any, to close and records closing
costs as stores are closed or identified to be closed. The Company closed one
store in the first quarter of fiscal 1997 and anticipates closing two stores in
the second quarter of fiscal 1997.

         Management believes that the Company's working capital, bank line of
credit and cash flows from operating activities will be sufficient to meet the
Company's operating and capital expenditure requirements through the end of
fiscal 1997.

INFLATION

         The Company does not believe that inflation has had a material effect
on the results of operations during the past three years. There can be no
assurance that the Company's business will not be affected by inflation in the
future.

NEW ACCOUNTING PRONOUNCEMENTS

         In February 1997, the Financial Accounting Standards Board issued
Statement of Financial Accounting Standards No. 128, "Earnings Per Share" ("SFAS
No. 128") which is effective for financial statements issued for periods ending
after December 15, 1997. SFAS No. 128 requires the disclosure of basic and
diluted earnings per share. For the thirteen weeks ended May 4, 1997, the amount
reported as net income per common and common equivalent share is not materially
different from that which would have been reported for basic and diluted
earnings per share in accordance with SFAS No. 128.


                                        8

<PAGE>   9
SEASONALITY AND QUARTERLY RESULTS

         The Company's business is seasonal by nature, with the Christmas and
back-to-school periods historically accounting for the largest percentage of
annual net sales. The Company's first quarter historically accounts for the
smallest percentage of annual net sales. In fiscal 1996 and fiscal 1995,
excluding net sales generated by new and expanded/relocated stores, the
Christmas and back-to-school periods together accounted for approximately 36% of
the Company's annual net sales and a higher percentage of the Company's
operating income. In fiscal 1996, excluding net sales generated by new and
expanded/relocated stores, approximately 43% of the Company's annual net sales
occurred in the first half of the fiscal year and 57% in the second half. The
Company's quarterly results of operations may also fluctuate significantly as a
result of a variety of factors, including the timing of store openings and the
amount of revenue contributed by new stores, the timing and level of markdowns,
the timing of store closings, expansions and relocations, competitive factors
and general economic conditions.

SAFE HARBOR STATEMENT

         The preceding "Management's Discussion and Analysis of Financial
Condition and Results of Operations" section contains certain forward-looking
statements within the meaning of Section 27A of the Securities Act of 1933 (the
"Securities Act") and Section 21E of the Securities Exchange Act of 1934 (the
"Exchange Act"), and the Company intends that such forward-looking statements be
subject to the safe harbors created thereby. These forward-looking statements
include the plans and objectives of management for future operations, including
plans and objectives relating to the future economic performance of the Company.
The forward-looking statements and associated risks set forth herein may include
or relate to: (i) the planned opening of approximately 39 stores and expansion
or relocation of 13 stores during the remainder of fiscal 1997; (ii) the
anticipated closing of two stores in the second quarter of fiscal 1997 and;
(iii) sufficiency of the Company's working capital, bank line of credit and cash
flows from operating activities for the Company's future operating and capital
requirements.

         The forward-looking statements are further qualified by important
factors that could cause actual results to differ materially from those in the
forward-looking statements, including, without limitation, the following: (i)
the ability of the Company to locate and obtain favorable store sites, negotiate
acceptable lease terms, and hire and train employees; (ii) management's ability
to manage the Company's planned expansion; (iii) the availability of merchandise
from the Company's vendors and private brand sources; (iv) the effect of
economic conditions; and (v) the effect of competitive pressures from other
retailers, particularly including those in the recently introduced juniors and
footwear categories. Results actually achieved thus may differ materially from
expected results in these statements.


                                        9

<PAGE>   10
                            PART II-OTHER INFORMATION

Item 1 - Legal Proceedings - Not Applicable

Item 2 - Changes in Securities - Not Applicable

Item 3 - Defaults Upon Senior Securities - Not Applicable

Item 4 - Submission of Matters to a Vote of Security Holders - Not Applicable

Item 5 - Other Information - Not Applicable

Item 6 - Exhibits and Reports on Form 8-K

         (a) Exhibits:

             (27) Financial Data Schedule

         (b) Reports on Form 8-K:

             No reports were filed on Form 8-K during the quarter for which 
             this report is filed.


                                       10

<PAGE>   11
                                   SIGNATURES

         Pursuant to the requirements of the Securities Exchange Act of 1934,
the registrant has duly caused this report to be signed on its behalf by the
undersigned thereunto duly authorized.

                                            Pacific Sunwear of California, Inc.
                                            (Registrant)


Date: May 23, 1997                           /s/ Greg H. Weaver
                                             -------------------------------
                                                 Greg H. Weaver 
                                                 Chief Executive Officer
                                                 and Director


Date: May 23, 1997                           /s/ Carl W. Womack
                                             -------------------------------
                                                 Carl W. Womack
                                                 Senior Vice President, Chief
                                                 Financial Officer and Secretary


                                       11