<PAGE>   1

                       SECURITIES AND EXCHANGE COMMISSION
                             Washington, D.C. 20549
                                    FORM 10-K


(Mark One)
[X]      ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES
         EXCHANGE ACT OF 1934 (NO FEE REQUIRED, EFFECTIVE OCTOBER 7, 1996)

For the fiscal year ended               December 31, 1996
                         ------------------------------------------------

[ ]      TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES
         EXCHANGE ACT OF 1934 (NO FEE REQUIRED)

For the transition period from                      to
                                -------------------     -----------------------

Commission file number              0-15956
                       ---------------------------------

                           Bank of Granite Corporation
--------------------------------------------------------------------------------
             (Exact name of registrant as specified in its charter)

         North Carolina                                56-1550545
----------------------------------        --------------------------------------
(State or other jurisdiction of            (I.R.S. Employer Identification No.)
 incorporation or organization)

P.O. Box 128, Granite Falls, N.C.                              28630
----------------------------------------         -------------------------------
(Address of principal executive offices)                     (Zip Code)

Registrant's telephone number, including area code     (704) 496-2000
                                                    ----------------------

           Securities registered pursuant to Section 12(b) of the Act:

   Title of each class              Name of each exchange on which registered
      Common Stock                                    NASDAQ
------------------------          ----------------------------------------------

           Securities registered pursuant to Section 12(g) of the Act:

                          Common Stock, $1.00 par value
                          -----------------------------
                                (Title of Class)

         Indicate by check mark if disclosure of delinquent filers pursuant to
Item 405 of Regulation S-K is not contained herein, and will not be contained,
to the best of registrant's knowledge, in definitive proxy or information
statements incorporated by reference in Part III of this Form 10-K or any
amendment to this Form 10-K. [X]

         Indicate by check mark whether the registrant (1) has filed all reports
required to be filed by Section 13 or 15(d) of the Securities Exchange Act of
1934 during the preceding 12 months (or for such shorter period that the
registrant was required to file such reports), and (2) has been subject to such
filing requirements for the past 90 days.  Yes   X    No
                                               -----     -----

         The aggregate market value of the voting stock held by non-affiliates
of the Registrant as of March 11, 1996, was $270,971,460.

Documents Incorporated by Reference

         PARTS I AND II: Annual Report to Shareholders for the fiscal year ended
December 31, 1996 (with the exception of those portions which are specifically
incorporated by reference in this Form 10-K, and the Annual Report to
Shareholders is not deemed to be filed as part of this report).

         PART III: Definitive Proxy Statement dated March 14, 1996 as filed
pursuant to Section 14 of the Securities Exchange Act of 1934 in connection with
the 1996 Annual Meeting of Shareholders.






<PAGE>   2



FORM 10-K CROSS-REFERENCE INDEX

<TABLE>
<CAPTION>
                                                                      ANNUAL            PROXY
                                                                      REPORT            REPORT
                                                                      1996              1997
                                                                      PAGE              PAGE

<S>                                                                   <C>               <C>
PART I

Item 1   Business                                                     None
         Executive Officers of the Registrant                         None
Item 2   Properties                                                   None
Item 3   Legal Proceedings                                            None
item 4   Submission of Matters to a Vote of Security Holders

PART II

Item 5   Market for the Registrant's Common Equity and Related
         Shareholder Matters                                          2 & 18            I-3
Item 6   Selected Financial Data                                          14
Item 7   Management's Discussion and Analysis of Financial
         Condition and Results of Operation                                             II-1  - II-11
Item 8   Financial Statements and Supplementary Data                                    II-12 - II-35
Item 9   Changes in and Disagreements with Accountants on
         Accounting and Financial Disclosure                          None

PART III

Item 10  Directors and Executive Officers of the Registrant                             I-10
Item 11  Executive Compensation                                                         I-11 - I-16
Item 12  Security Ownership of Certain Beneficial Owners and
         Management                                                                     I-3  & I-10
Item 13  Certain Relationships and Related Transactions                                 I-18

PART IV

Item 14  Exhibits, Financial Statement Schedules and Reports                            II-12 - II-35
         on Forms 8-K
</TABLE>







<PAGE>   3

                                     PART I


ITEM 1 - BUSINESS

Bank of Granite Corporation (the "Registrant") is a Delaware Corporation
organized June 1, 1987 as a holding company. The Registrant currently engages in
no operations other than ownership and operation of Bank of Granite (the
"Bank"), a state bank chartered under the laws of North Carolina in 1906. The
Registrant conducts its business from eleven banking offices located in
Caldwell, Catawba, and Burke counties in North Carolina. As of December 31,
1996, the Bank was the ninth largest bank in North Carolina.

GENERAL BUSINESS

The Bank's principal activities include the taking of demand and time deposits
and the making of loans, secured and unsecured, to individuals, associations,
partnerships, and corporations. Bank of Granite is an independent community
bank. The majority of its customers are individuals and small businesses. No
material part of its business is dependent upon a single customer or a few
customers whose loss would have an adverse effect on the business of the Bank.
No material portion of the business of the Bank is seasonal.

TERRITORIAL SERVED AND COMPETITION

The Bank now operates banking offices in Granite Falls, Lenoir, and the Hibriten
and Whitnel sections of Lenoir, Hudson, Newton, Morganton, Hickory, and the
Springs Road, Viewmont and Long View sections of Hickory for a total of eleven
offices. Banking laws of North Carolina allow statewide branching, resulting in
commercial banking in the state being extremely competitive.

There are six other commercial banks located in the Bank's service area in
Caldwell County and one credit union. The most recent FDIC figures available
indicate that Bank of Granite has funds on deposit amounting to 24% of the total
commercial bank deposits in Caldwell County as of June 30, 1996.

There are nine other commercial banks and eight credit unions located in the
Bank's service area in Catawba County. The most recent FDIC figures available
show that Bank of Granite's deposits comprise 14% of the total commercial bank
deposits in Catawba as of June 30, 1996.

On July 10, 1989 Bank of Granite opened a full service office in Morganton.
Competing in the same service area are five other commercial banks, five credit
unions, and one savings and loan bank. The most recent FDIC figures available
show that Bank of Granite's deposits comprise 4% of the total commercial bank
deposits in Burke County as of June 30, 1996.

EMPLOYEES

As of December 31, 1996, Bank of Granite had 170 full-time equivalent employees.
The Registrant considers its relationship with its employees to be excellent.


                                                                               1

<PAGE>   4

SUPERVISION AND REGULATION

The following summaries of statutes and regulations affecting bank holding
companies and banks do not purport to be complete. Such summaries are qualified
in their entirety by reference to such statutes and regulations.

The Bank Holding Company Act

The Registrant is a bank holding company within the meaning of the Bank Holding
Company Act of 1956, as amended (the "Bank Holding Company Act"), and is
required to register as such with the Board of Governors of the Federal Reserve
System (the "Federal Reserve Board").

A bank holding company is required to file with the Federal Reserve Board annual
reports and other information regarding its business operations and those of its
subsidiaries. It is also subject to examination by the approval prior to
acquiring , directly or indirectly, more than 5% of the voting stock of such
bank, unless it already owns a majority of the voting stock of such bank.
Furthermore, a bank holding company is, with limited exceptions, prohibited from
acquiring direct or indirect ownership or holding company, and must engage only
in the business of banking or managing or controlling banks or furnishing
services to or performing services for its subsidiary banks. One of the
exceptions to this prohibition is the ownership of shares of a company the
activities of which the Federal Reserve Board has determined to be so closely
related to banking or managing or controlling banks as to be a proper incident
thereto.

The Federal Reserve Board has cease-and-desist powers over parent bank holding
companies and non-banking subsidiaries where their action would constitute a
serious threat to the safety, soundness or stability of a subsidiary bank.

While the Registrant is not presently subject to any regulatory restrictions on
dividends, the Registrant's ability to pay dividends will depend to a large
extent on the amount of dividends paid by the Bank and any other subsequently
acquired entities. The Bank, as a North Carolina banking corporation, may pay
dividends only out of undivided profits as determined pursuant to North Carolina
General Statutes Section 53-87. As of December 31, 1996, the Bank had undivided
profits of approximately $70.0 million. Additionally, current federal
regulations require that the Bank maintain a ratio of total capital to assets,
as defined by regulatory authorities, in excess of 6%. As of December 31, 1996,
this ratio was 16.7%, leaving approximately $40.8 million of the Bank's
undivided profits available for the payment of dividends. The Bank is, and such
other entities may be subject to regulatory restrictions on the payment of
dividends.

In an effort to achieve a measurement of capital adequacy that is more sensitive
to the individual risk profiles of financial institutions, the various financial
institution regulators mandate minimum capital regulations and guidelines that
categorize various components of capital and types of assets and measure capital
adequacy in relation to a particular institution's relative levels of those
capital components and the level of risk associated with various types of assets
of that financial institution. The FDIC and the FRB statements of policy on
"risk-based capital" require the Company to maintain a level of capital
commensurate with the risk profile assigned to its assets in accordance with the
policy statements. The capital standards call for minimum total capital of 8
percent of risk-adjusted assets. At December 31, 1996, the Company's tier 1
ratio and capital ratio to risk-adjusted assets was 23.4% and 24.6%
respectively. The Company's leverage ratio at December 31, 1996 was 17.6%. The
Company is in compliance with all regulatory capital requirements.

The Company cannot predict what other legislation might be enacted or what other
regulation might be adopted or, if enacted or adopted, the effect thereof.

The Bank is subject to supervision and regulation, of which regular bank
examinations are a part, by the Federal Deposit Insurance Corporation and North
Carolina State Banking Commission. The Bank is a member of the Federal Deposit
Insurance Corporation (the "FDIC"), which currently insures the deposits of each
member bank to a maximum of $100,000 per depositor. For this protection, each
bank pays a semi-annual statutory assessment and is subject to the rules and
regulations of the FDIC.

Federal banking laws applicable to all depository financial institutions, among
other things, (I) afford federal bank regulatory agencies with powers to prevent
unsafe and unsound banking practices; (II) restrict preferential loans by bands
to "insiders" of banks; (III) require banks to keep information on loans to
major shareholders and executive officers, and (IV) bar certain directory

                                                                               2


<PAGE>   5

and officer interlocks between financial institutions. The prohibitions against
preferential loans and certain director and officer interlocks may inhibit the
ability of the Bank and the Registrant to obtain experienced and capable
officers and directors, to replace presently proposed officers and directors, or
to add to their number.

The registrant is an "affiliate" of the Bank within the meaning of the Federal
Reserve Act, which imposes restrictions on loans by the Bank to the Registrant,
on investments by the Bank in the stock or securities of the security for loans
by the Bank to any borrower. The Registrant is also subject to certain
restrictions with respect to engaging in the business of issuing, underwriting
and distributing securities.

Shareholders of banks (including bank holding companies which own stock in
banks) may be compelled by bank regulatory authorities to invest additional
capital in the event their banks experience either significant loan losses or
rapid growth of loans or deposits. In addition, the Registrant may also be
required to provide additional capital to any additional banks which it acquires
as a condition to obtaining the approvals and consents of regulatory authorities
in connection with such acquisitions.

Effects of Governmental Monetary Policy and Economic Controls

The Registrant is directly affected by governmental monetary policy and by
regulatory measures affecting the banking industry in general. Of primary
importance is the Federal Reserve Board, whose actions directly affect the money
supply and, in general, affect banks' lending abilities by increasing or
decreasing the cost and availability of bank credit in order to combat recession
and curb inflationary pressures in the economy by open market operations in the
United States government securities, changes in the discount rate on the member
bank borrows, and changes in reserve requirements against bank deposits.

Deregulation of interest rates paid by banks on deposits and the types of
deposits that may be offered by banks have eliminated minimum balance
requirements and rate ceilings on various types of time deposit accounts. The
effect of these specific actions and, in general, the deregulation of deposit
interest rates have increased banks' costs of funds and made the more sensitive
to fluctuations in money market rates.

In view of changing conditions in the national economy and money markets, as
well as the effect of actions by monetary and fiscal authorities, no prediction
can be made as to possible future changes in interest rates, deposit levels,
loan demand or the business and earnings of the Registrant.

ITEM 2 - PROPERTIES

Bank of Granite owns all of its properties which are listed below.

GRANITE FALLS, NC

Granite Falls office (home office) - 8,735 square foot building located on a 1.2
acre lot.

A masonry and frame storage building containing 735 square feet with an
adjoining 100' x 221' lot.

Operations Center - 70' x 168' building containing 11,769 square foot which
located on a 1.05 acre lot.

15' x 25' building which houses our print shop.

LENOIR, NC

Lenoir office - building is approximately 7,400 square feet.

Whitnel office - building is approximately 2,530 square feet located on a lot
containing 45,500 square feet.

Hibriten office - building is approximately 2,480 square feet situated on 2.10
acres of land.


                                                                               3

<PAGE>   6

HUDSON, NC

Hudson office - building is approximately 4,235 square feet located on a 4.10
acre lot.

HICKORY, NC

Hickory office - 54' x 81' structure containing 9,515 square feet situated on a
100' x 200' lot.

Bank of Granite Plaza - two story building with two sections. One section
contains approximately 7,520 square feet, the other sections contains 7,572
square feet.

Newton office - 43' x 84' structure located on a 200' x 200' lot.

Springs Road office - 43' x 84' structure located on a 1.6 acre lot.

Viewmont office - approximately 4,200 square feet located on a two acre lot.

Longview office - 2,440 square feet brick building located on a corner lot
containing 1.1 acres.

MORGANTON, NC

Morganton office contains approximately 5,400 square feet situated on a lot
containing .78 acre.

ITEM 3 - LEGAL PROCEEDINGS

There were no significant legal proceedings outstanding at December 31, 1995.

ITEM 4 - SUBMISSION OF MATTERS TO A VOTE OF SECURITY HOLDERS

There were no matters submitted to a vote of shareholders in the Fourth Quarter
of the Company's fiscal year.







                                                                               4

<PAGE>   7

                                     PART II

ITEM 5 - MARKET FOR THE REGISTRANT'S COMMON EQUITY AND RELATED
         STOCKHOLDER'S MATTERS

The information required by this item is set forth on Pages 2 and 18 on the
Company's 1996 ANNUAL REPORT TO SHAREHOLDERS under the headings "Market and
Dividend Summary" and "Shareholder Information", and respectively, on Page I-3
of the 1997 PROXY STATEMENT under the heading "Principal Holders of Voting
Securities". The above information is incorporated by reference.

ITEM 6 - SELECTED FINANCIAL DATA

The information required by this item is set forth on Page 14 in the Company's
1996 ANNUAL REPORT TO SHAREHOLDERS under the heading "Selected Financial Data"
of which information is incorporated herein by reference.

ITEM 7 - MANAGEMENT'S DISCUSSION AND ANALYSIS OF FINANCIAL
         CONDITION AND RESULTS OF OPERATIONS

The information required by this item is set forth on Pages II-1 through II-11 
in the Company's 1997 PROXY STATEMENT for the heading "Management's Discussion
and Analysis", which is incorporated herein by reference.

ITEM 8 - FINANCIAL STATEMENTS & SUPPLEMENTARY DATA

The Consolidated financial statements, and notes thereto and the independent
auditors' report on Pages II-12 through II-35 of the 1997 PROXY STATEMENT
for the year ended December 31, 1996 are incorporated herein by reference.

ITEM 9 - CHANGES IN AND DISAGREEMENT WITH ACCOUNTANTS ON ACCOUNTING
         AND FINANCIAL DISCLOSURE

There has been no disagreement with accountants on accounting and financial
disclosure as defined by Item 304 of Regulation S-K.



                                                                               5

<PAGE>   8

                                    PART III

ITEM 10 - DIRECTORS AND EXECUTIVE OFFICERS OF THE REGISTRANT

The information required by this item is set forth on Page I-10 under the
heading "Directors and Executive Officers of Bank of Granite Corporation" in the
definitive proxy materials of the Company filed in connection with its 1997
ANNUAL MEETING OF SHAREHOLDERS. The information required by this item contained
in such definitive proxy materials is incorporated herein by reference.

ITEM 11 - EXECUTIVE COMPENSATION

The information required by this item is set forth on Pages I-11 through I-16 
in the definitive proxy materials of the Company filed in connection with its
1997 ANNUAL MEETING OF SHAREHOLDERS, which information is incorporated herein by
reference.

ITEM 12 - SECURITY OWNERSHIP OF CERTAIN BENEFICIAL OWNERS AND
          MANAGEMENT

The information required by this item is set forth on Pages I-3 and I-10 in
the definitive proxy materials of the Company filed in connection with its 1997
ANNUAL MEETING OF SHAREHOLDERS, which information is incorporated herein by
reference.

ITEM 13 - CERTAIN RELATIONSHIPS AND RELATED TRANSACTIONS

The information required by this item is set forth on Page I-18 under the
heading "Transactions with Officers and Directors" in the definitive proxy
materials of the Company filed in connection with its 1997 ANNUAL MEETING OF
SHAREHOLDERS, which information is incorporated herein by reference.






                                                                               6

<PAGE>   9

                                     PART IV

1.       Financial Statements
         The information required by this item is set forth on Pages II-12
         through II-35 in the Company's 1997 PROXY STATEMENT TO SHAREHOLDERS,
         which is incorporated herein by reference.

2.       Financial Statement Schedules
         None

13.      Annual Report to Security Holders

21.      Subsidiaries of the Registrant
         Bank of Granite Corporation has one subsidiary, Bank of Granite,
         incorporated in 1906 and operates in North Carolina

27.      Financial Data Schedule (for SEC use only)

B.       No reports on Form 8-K were filed during the year.










                                                                               7

<PAGE>   10


                                   SIGNATURES

Pursuant to the requirements of Section 13 or 15(d) of the Securities Exchange
Act of 1934, the registrant has duly caused this report to be signed on behalf
by the undersigned, thereunto duly authorized.

BANK OF GRANITE CORPORATION

By:      s/John A. Forlines, Jr.
         ------------------------------------
         John A. Forlines, Jr.
         Chairman and Chief Executive Officer

Pursuant to the requirements of the Securities and Exchange Act of 1934, this
report has been signed below by the following persons on behalf of the
registrant and in the capacities and on the dates indicated.

       Signature                         Title                        Date

s/John A. Forlines, Jr.            Chairman and Chief            March 10, 1997
-----------------------            Executive Officer
John A. Forlines, Jr.              

s/Randall C. Hall                  Vice President and            March 10, 1997
-----------------------            Chief Financial       
Randall C. Hall                    Officer and Principal 
                                   Accounting Officer    
                                   

s/John N. Bray                     Director                      March 10, 1997
-----------------------
John N. Bray

s/Robert E. Cline                  Director                      March 10, 1997
-----------------------
Robert E. Cline

s/John A. Forlines, Jr.            Director                      March 10, 1997
-----------------------
John A. Forlines, Jr.

s/Barbara F. Freiman               Director                      March 10, 1997
-----------------------
Barbara F. Freiman

s/Myron L. Moore, Jr.              Director                      March 10, 1997
-----------------------
Myron L. Moore, Jr.

s/Charles M. Snipes                Director                      March 10, 1997
-----------------------
Charles M. Snipes

s/Boyd C. Wilson, Jr.              Director                      March 10, 1997
-----------------------
Boyd C. Wilson, Jr.







                                                                               8