<PAGE>   1

                       SECURITIES AND EXCHANGE COMMISSION
                             Washington, D.C. 20549

                              ---------------------

                                    FORM 10-Q

(Mark One)

[X]  QUARTERLY REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES 
     EXCHANGE ACT OF 1934

For the quarterly period ended September 30, 1997

[ ]  TRANSACTION REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES 
     EXCHANGE ACT OF 1934

For the transition period from _____ to _____

                           Commission File No. 0-20698

                          BROOKTROUT TECHNOLOGY, INC. 
             ------------------------------------------------------
             (Exact name of registrant as specified in its charter)


              MASSACHUSETTS                         04-2814792
              (State or other                       (I.R.S. employer
              jurisdiction of                       identification
              incorporation or                      number)
              organization)

                       410 First Avenue
                    NEEDHAM, MASSACHUSETTS                          02194 .
            ----------------------------------------              ---------- 
            (Address of principal executive offices)              (Zip code)

                          Registrant's telephone number
                       including area code: (781) 449-4100

        Indicate by check mark whether the registrant: (1) has filed all reports
required to be filed by Section 13 or 15(d) of the Securities Exchange Act of
1934 during the preceding 12 months (or for such shorter period that the
registrant was required to file such reports), and (2) has been subject to such
filing requirements for the past 90 days.
Yes   X    No 
    -----     -----

        As of November 1, 1997, 10,710,564 shares of Common Stock, $.01 par
value per share, were outstanding.




                               Page 1 of 16 pages
                        Exhibit Index Appears on Page 14
<PAGE>   2

                           BROOKTROUT TECHNOLOGY, INC.

                                    FORM 10-Q

                    FOR THE QUARTER ENDED SEPTEMBER 30, 1997

                                TABLE OF CONTENTS

                                                                          Page
                                                                          ----

PART I     FINANCIAL INFORMATION

Item 1.    Condensed Consolidated Financial Statements

           Condensed Consolidated Balance Sheets as of
           September 30, 1997 and December 31, 1996                         3

           Condensed Consolidated Statements of Income for
           the Three Months Ended September 30, 1997 and
           September 30, 1996, and the Nine Months Ended
           September 30, 1997 and September 30, 1996                        4

           Condensed Consolidated Statements of Cash Flows
           for the Nine Months Ended September 30, 1997 and
           September 30, 1996                                               5

           Notes to Condensed Consolidated Financial
           Statements                                                       6

Item 2.    Management's Discussion and Analysis of Financial
           Condition and Results of Operations


           Three Months Ended September 30, 1997 and 1996                   9

           Nine Months Ended September 30, 1997 and 1996                   10

           Liquidity and Capital Resources                                 11

PART II    OTHER INFORMATION

Item 1.    Legal Proceedings                                               12

Item 6.    Exhibits                                                        12

           Signatures                                                      13

           Exhibit Index                                                   14



<PAGE>   3

                   BROOKTROUT TECHNOLOGY, INC.
              Condensed Consolidated Balance Sheets
                (In thousands, except share data)
                           (Unaudited)

<TABLE>
<CAPTION>
                                                               September 30,    December 31,
                                                                   1997             1996
                                                                   ----             ----
<S>                                                               <C>              <C>    

ASSETS

  Current assets:
    Cash and equivalents ......................................   $23,313          $30,738
    Marketable securities .....................................     8,267            8,976
    Accounts receivable (less allowance for doubtful
      accounts of $1,270 in 1997 and $524 in 1996) ............    10,673            7,107
    Inventory .................................................     8,690            5,504
    Other current assets ......................................     3,663            1,625
                                                                  -------          -------
      TOTAL CURRENT ASSETS ....................................    54,606           53,950
                                                                  -------          -------

  Equipment and furniture:
    Computer equipment ........................................     5,900            2,822
    Furniture and office equipment ............................     3,290            2,476
                                                                  -------          -------
      Total ...................................................     9,190            5,298
      Less accumulated depreciation and amortization ..........    (2,553)          (1,438)
                                                                  -------          -------

      EQUIPMENT AND FURNITURE - NET ...........................     6,637            3,860

  Investment and other assets .................................     1,261              556
                                                                  -------          -------

          TOTAL ...............................................   $62,504          $58,366
                                                                  =======          =======

LIABILITIES AND STOCKHOLDERS' EQUITY

  Current liabilities:
    Accounts payable and other accruals .......................   $11,467          $ 8,342
    Accrued compensation and commission .......................     2,253            1,996
    Accrued income taxes ......................................        --              204
                                                                  -------          -------
      TOTAL CURRENT LIABILITIES ...............................    13,720           10,542

  Deferred rent ...............................................       209              232

  Stockholders' equity:
    Common stock, $.01 par value; authorized, 25,000,000
      shares; issued and outstanding 10,709,101 shares in
      1997 and 10,683,352 in 1996 .............................       107              107
    Additional paid-in capital ................................    31,848           31,785
    Unrealized gains (losses) on marketable securities ........        11               (8)
    Retained earnings .........................................    16,609           15,708
                                                                  -------          -------

    STOCKHOLDERS' EQUITY ......................................    48,575           47,592
                                                                  -------          -------

    TOTAL .....................................................   $62,504          $58,366
                                                                  =======          =======
</TABLE>




See notes to condensed consolidated financial statements.




<PAGE>   4

                   BROOKTROUT TECHNOLOGY, INC.
           Condensed Consolidated Statements of Income
              (In thousands, except per share data)

                           (Unaudited)

<TABLE>
<CAPTION>
                                                        Three Months Ended        Nine Months Ended
                                                           September 30,            September 30,
                                                         1997        1996         1997         1996
                                                         ----        ----         ----         ----
<S>                                                    <C>          <C>          <C>          <C>    

REVENUE .............................................  $19,493      $15,874      $49,288      $40,619  
                                                       -------      -------      -------      -------  
                                                                                                       
Cost and expenses:                                                                                     
  Cost of product sold ..............................    8,804        7,148       21,869       17,953  
  Research and development ..........................    3,990        1,799        9,280        5,047  
  Selling, general and administrative ...............    5,679        3,558       14,192        9,796  
  Acquired research and development .................       --           --        3,746           --  
  Acquisition related costs .........................       --           --           --        1,236  
                                                       -------      -------      -------      -------  
                                                                                                       
      Total cost and expenses .......................   18,473       12,505       49,087       34,032  
                                                       -------      -------      -------      -------  
                                                                                                       
INCOME FROM OPERATIONS ..............................    1,020        3,369          201        6,587  
                                                                                                       
Interest income, net ................................      327          328        1,249          837  
                                                       -------      -------      -------      -------  
                                                                                                       
Income before income tax provision ..................    1,347        3,697        1,450        7,424  
                                                                                                       
Income tax provision ................................      531        1,479          548        3,346  
                                                       -------      -------      -------      -------  
                                                                                                       
NET INCOME ..........................................  $   816      $ 2,218      $   902      $ 4,078  
                                                       =======      =======      =======      =======  
                                                                                                       
NET INCOME PER COMMON AND COMMON EQUIVALENT SHARE ...  $  0.07      $  0.20      $  0.08      $  0.38  
                                                       =======      =======      =======      =======  
                                                                                                       
Weighted average number of common and                              
  common equivalent shares outstanding ..............   11,230       11,135       11,294       10,684
                                                       =======      =======      =======      =======
</TABLE>





See notes to condensed consolidated financial statements.


<PAGE>   5

                   BROOKTROUT TECHNOLOGY, INC.
         Condensed Consolidated Statements of Cash Flows
                          (In thousands)
                           (Unaudited)

<TABLE>
<CAPTION>
                                                                         Nine Months Ended
                                                                            September 30,
                                                                       -----------------------
                                                                         1997           1996
                                                                       --------       --------
<S>                                                                    <C>            <C>    

ASH FLOWS FROM OPERATING ACTIVITIES:
   Net income ........................................................ $    902       $ 4,078
   Adjustments to reconcile net income to cash provided
      by operating activities:
         Deferred income taxes .......................................   (3,618)         (259)
         Depreciation and amortization ...............................    1,115           387
         Acquired research and development ...........................    3,746            --
         Amortization of net premium (discount) on
            marketable securities ....................................      (37)           27
         Increase (decrease) in cash, excluding the effects of an
            acquisition, from:
               Accounts receivable ...................................      214          (780)
               Inventory .............................................      267        (1,394)
               Other current assets ..................................    1,381           (36)
               Accounts payable and accrued expenses .................      349         2,784
                                                                       --------       -------

                 Cash provided by operating activities ...............    4,319         4,807

CASH FLOWS FROM INVESTING ACTIVITIES:
   Expenditures for equipment and furniture ..........................   (2,029)       (2,652)
   Acquisition of Netaccess (net of cash acquired) ...................  (10,542)           --
   Purchases of marketable securities ................................   (6,808)       (1,801)
   Maturities and sales of marketable securities .....................    7,572         2,386
                                                                       --------       -------

                 Cash provided by (used in) investing activities .....  (11,807)       (2,067)
                                                                       --------       -------

CASH FLOWS FROM FINANCING ACTIVITIES:
   Proceeds from the sale of common stock ............................       63        11,108
   Distributions to stockholders .....................................       --           (12)
   Repayment of long-term debt .......................................       --            (6)
                                                                       --------       -------

                 Cash provided by financing activities ...............       63        11,090
                                                                       --------       -------

INCREASE (DECREASE) IN CASH AND EQUIVALENTS ..........................   (7,425)       13,830
CASH AND EQUIVALENTS, BEGINNING OF PERIOD ............................   30,738        14,230
                                                                       --------       -------

CASH AND EQUIVALENTS, END OF PERIOD .................................. $ 23,313       $28,060
                                                                       ========       =======

</TABLE>



See notes to condensed consolidated financial statements.



<PAGE>   6

BROOKTROUT TECHNOLOGY, INC.

Notes to Condensed Consolidated Financial Statements

1.      Basis of Presentation

        The accompanying unaudited condensed consolidated financial statements
have been prepared by the Company pursuant to the rules and regulations of the
Securities and Exchange Commission regarding interim financial reporting. They
should be read in conjunction with the audited consolidated financial statements
incorporated by reference in or included in the Company's 1996 Annual Report on
Form 10K.

        In the opinion of management, the accompanying unaudited condensed
consolidated financial statements have been prepared on the same basis as the
audited consolidated financial statements and include all adjustments,
consisting only of normal recurring adjustments, necessary for a fair
presentation of the interim periods presented.

        The operating results for the interim periods presented are not
necessarily indicative of the results which could be expected for the full year.


2.      Business Combination

        On June 30, 1997, the Company acquired the assets and assumed certain
liabilities of Netaccess, Inc., a worldwide supplier of Primary Rate ISDN
network interface products and multiport modem products for open,
standards-based remote access and computer telephony systems. The purchase price
was $10.5 million, paid in cash, and the Company agreed to assume certain
liabilities aggregating $2 million.

        The acquisition has been accounted for as a purchase, and accordingly,
the results of operations of Netaccess, Inc. have been included in the Company's
financial statements from the date of acquisition. The purchase price has been
allocated to the assets and other items acquired based upon their fair values.
The allocation reflected in these financial statements is preliminary and may
change upon completion of independent appraisals and other matters.

        Based upon preliminary appraisals, the Company has recorded a charge of
$3.7 million ($2.3 million, net of tax benefits) representing the estimated
value of Netaccess' research and development efforts in process. Such efforts
had not yet reached technological feasibility and did not possess alternative
uses. Costs to complete the effort on these products is estimated to be $4.4
million, which will be expensed in the normal course of the Company's ongoing
product development program.



<PAGE>   7

3.      Net income per share

        Net income per common and per common equivalent share are computed using
the weighted average number of common and dilutive common equivalent shares
outstanding during the period, using the provisions of Accounting Principles
Board Opinion No. 15, "Earnings per Share." Dilutive common equivalent shares
represent shares issuable upon exercise of stock options, calculated using the
treasury stock method.

        In March 1997, the Financial Accounting Standards Board released
Statement of Financial Accounting Standards (SFAS) No. 128, "Earnings per
Share," which the Company will adopt in the fourth quarter of 1997. Had SFAS No.
128 been effective for the three months ended September 30, 1997 and 1996, and
for the nine months ended September 30, 1997 and 1996, reported earnings per
share on a pro forma basis would have been as follows:

<TABLE>
<CAPTION>
                       Three Months Ended           Nine Months Ended
                          September 30,                September 30,
                       1997          1996           1997         1996
                       ----          ----           ----         ----
<S>                    <C>          <C>             <C>         <C>  

Basic ...............  $0.08        $0.22           $0.08       $0.42

Diluted .............  $0.07        $0.20           $0.08       $0.38

</TABLE>


4.      Inventory

        Inventory is carried at the lower of cost (first-in, first-out basis) or
market and consisted of the following:

<TABLE>
<CAPTION>
                                   September 30,          December 31,
                                       1997                  1996
                                       ----                  ----

        <S>                         <C>                   <C>       
        Raw materials ............  $3,251,000            $3,740,000
        Work in process ..........   2,769,000             1,104,000
        Finished goods ...........   2,670,000               660,000
                                    ----------            ----------
           Total .................  $8,690,000            $5,504,000
                                    ==========            ==========
</TABLE>
              

5.      Major Customers

        One customer accounted for approximately 31% and 38% of net revenue for
the three months ended September 30, 1997 and 1996, respectively, and 32% and
31% for the nine months ended September 30, 1997 and 1996, respectively.




<PAGE>   8

6.      Marketable Securities

        Marketable securities consist mainly of U.S. government securities
purchased with remaining maturities in excess of three months.

        The amortized cost of these securities at September 30, 1997 was
$8,256,000. Net unrealized holding gains were $11,300 at September 30, 1997.


7.      Income Taxes

        A reconciliation of the statutory federal rate to the effective rate is
as follows:


<TABLE>
<CAPTION>
                                     Three Months Ended          Nine Months Ended
                                        September 30,               September 30,
                                     1997          1996          1997         1996
                                     ----          ----          ----         ----
<S>                                  <C>            <C>           <C>         <C>

Statutory tax rate                   34%            34%           34%         34%
State taxes, net of
     federal benefit                  5              6             4           6
Other                                --             --            --           5
                                     --             --            --          --

Effective tax rate                   39%            40%           38%         45%
                                     ==             ==            ==          ==

</TABLE>

8.      International Sales

        International sales, principally exported from the United States,
accounted for approximately 17% and 16% of revenue for the three months ended
September 30, 1997 and 1996, respectively, and 19% and 18% for the nine months
ended September 30, 1997 and 1996, respectively.





<PAGE>   9

ITEM 2. MANAGEMENT'S DISCUSSION AND ANALYSIS OF FINANCIAL CONDITION AND RESULTS
OF OPERATIONS.


RESULTS OF OPERATIONS


Three Months Ended September 30, 1997 and 1996

        Revenue during the three months ended September 30, 1997 increased by
approximately 23% to $19,493,000, up from $15,874,000 during the three months
ended September 30, 1996. This growth was primarily attributable to an increase
in Primary Rate ISDN telephone network interface products due to the acquisition
of Netaccess, Inc. on June 30, 1997. In addition, there were increased shipments
of Show N Tel software development tools.

        Cost of product sold was $8,804,000, or 45% of revenue, during the three
months ended September 30, 1997, compared to $7,148,000, or 45% of revenue, for
the same period in 1996, resulting in gross profit margins of approximately 55%
for both periods.

        Research and development expense was $3,990,000, or 20% of revenue,
compared with $1,799,000, or 11% of revenue, for the three months ended
September 30, 1997 and 1996, respectively. The increase was primarily
attributable to development work being performed by Netaccess, Inc. related to
their next generation of products and also reflects the Company's continuing
development efforts for its TR Series product family, computer telephony
development tools, Brooktrout Open Systems Telephony Architecture (BOSTON),
network interface products and Brooktrout Interspeed, as well as fax and OEM
systems development. The Company intends to continue to commit significant
resources to product development.

        Selling, general and administrative expense was $5,679,000 during the
three months ended September 30, 1997, compared with $3,558,000 during the same
period in 1996. This higher expense level resulted from increased staffing,
depreciation and rental expenses, primarily related to the acquisition of
Netaccess, Inc. As a percentage of revenue, selling, general and administrative
expense for the third quarter of 1997 was 29% of revenue, compared with 22% for
the third quarter of 1996.

        For the three months ended September 30, 1997, interest and other income
was $327,000, compared with $328,000 for the same period in 1996.

        The Company's effective tax rate was 39% in 1997 and 40% in 1996, based
on the Company's estimated effective tax rate for the full year.




<PAGE>   10


Nine Months Ended September 30, 1997 and 1996


        Revenue during the nine months ended September 30, 1997 increased by
approximately 21% to $49,288,000, up from $40,619,000 during the nine months
ended September 30, 1996. The growth was attributable to an increase in Primary
Rate ISDN telephone network interface products for computer telephony systems
and increased shipments of OEM voice systems and TR Series products. The
increase in the sale of network interface products resulted primarily from the
acquisition of Netaccess, Inc. The Company's voice mail business increased due
to shipments of a new integrated voice mail system. Increased sales from the TR
Series products reflect the growth in the local area network/fax market segment
served by the Company.

        Cost of product sold was $21,869,000, or 44% of revenue, during the nine
months ended September 30, 1997, compared to $17,953,000, or 44% of revenue, for
the same period in 1996, resulting in gross profit margins of approximately 56%
for both periods.

        Research and development expense was $9,280,000, or 19% of revenue,
compared with $5,047,000, or 12% of revenue, for the nine months ended September
30, 1997 and 1996, respectively. The increase reflects the Company's continuing
development efforts for its TR Series product family, computer telephony
development tools, Brooktrout Open Systems Telephony Architecture (BOSTON),
network interface products and Brooktrout Interspeed, as well as fax and OEM
systems development. In addition, research and development expenses increased
following the acquisition of Netaccess, Inc.

        Selling, general and administrative expense was $14,192,000 during the
nine months ended September 30, 1997, compared with $9,796,000 during the same
period in 1996. This higher expense level resulted from increased staffing, rent
and depreciation expenses, partly due to the acquisition of Netaccess, Inc. As a
percentage of revenue, selling, general and administrative expense for the first
nine months of 1997 was 29% of revenue, compared with 24% for the first nine
months of 1996.

        On June 30, 1997, the Company acquired the assets and assumed certain
liabilities of Netaccess, Inc., a worldwide supplier of Primary Rate ISDN
network interface products and multiport modem products for open,
standards-based remote access and computer telephony systems. The purchase price
was $10.5 million, paid in cash, and the Company also agreed to assume certain
liabilities aggregating $2 million. Based upon preliminary appraisals, the
Company has recorded a charge of $3.7 million ($2.3 million, net of tax
benefits) representing the estimated value of Netaccess' research and
development efforts in-process.

        For the nine months ended September 30, 1997, interest and other income
was $1,249,000, compared with $837,000 for the same period in 1996.

        The Company's effective tax rate, adjusted for significant permanent or
other differences , was 38% and 45% for the nine months ended September 30, 1997
and 1996, respectively. For the full year 1997, the Company expects the
effective tax rate to approximate 39%.




<PAGE>   11


Liquidity and Capital Resources

        For the nine months ended September 30, 1997, the Company funded its
operations principally through operating revenue. In August 1997, the Company
renewed its working capital line of credit. Under the renewed line of credit,
the Company may borrow up to $10,000,000 on an unsecured basis, all of which may
be used for issuance of letters of credit, subject to compliance with certain
covenants. The line of credit will expire in July 1998 and at that time any
outstanding balances would be payable in full. Any amounts borrowed under the
line would be subject to interest at the bank's prime rate. At September 30,
1997 there were no commitments outstanding on letters of credit; no borrowings
have been made during any period presented.

        The Company's working capital decreased from $43.4 million at December
31, 1996 to $40.9 million at September 30, 1997. The decrease in working capital
was caused by the payment of $10.5 million in cash to acquire the assets of
Netaccess. This decrease was offset , in part, by an increase in working capital
of approximately $4.8 million, representing the working capital of Netaccess
acquired by the Company. Other increases in working capital represented
increases in cash, receivables and inventory consistent with the positive cash
flow from operations and the growth in the business experienced during the nine
months ended September 30, 1997.

        During the first nine months of 1997, the Company invested approximately
$2 million in capital equipment. The Company currently has no material
commitments for additional capital expenditures.

        The Company anticipates that cash flows from operations, together with
current cash and marketable securities balances and funds available under the
Company's line of credit, will be sufficient to meet the Company's working
capital and capital equipment expenditure requirements for the foreseeable
future.

Recent Accounting Pronouncements

        In March 1997, the Financial Accounting Standards Board (FASB) issued
Statement of Financial Accounting Standards (SFAS) No. 128, "Earnings per
Share," which will be effective during the fourth quarter of 1997. SFAS No. 128
will require the Company to restate all previously reported earnings per share
information to conform with the new pronouncement's requirements.

        In June 1997, the FASB issued SFAS No. 131, "Disclosures about Segments
of an Enterprise and Related Information." This statement will be effective for
1998. SFAS No. 131 establishes standards for reporting information about
operating segments of the Company. This pronouncement would not have an impact
on reported net income or on the financial position of the Company.



<PAGE>   12

Part II. OTHER INFORMATION

Item 1. Legal Proceedings

        On October 4, 1996, Syntellect Technology Corp. ("Syntellect") filed a
Complaint against the Company in the United States District Court for the
Northern District of Texas, alleging infringement of certain patents held by
Syntellect relating to certain aspects of "automated attendant" technology.
Syntellect's Complaint does not identify the products of the Company which
allegedly infringe Syntellect's patents. The Complaint seeks injunctive relief,
damages in an unspecified amount, and multiple damages on account of alleged
willful infringement. A Motion to Dismiss the action, filed by the Company in
December 1996, was denied. On October 15, 1997, the Company filed a Motion for
Summary Judgment, which remains pending. The Company is reviewing the patents at
issue, and intends to defend the case vigorously.

Items 2. through 5.

        None

Item 6. Exhibits

        (a) Exhibits

                11.     Computation of earnings per share



<PAGE>   13



SIGNATURES

        Pursuant to the requirements of the Securities Exchange Act of 1934, the
registrant has duly caused this report to be signed on its behalf by the
undersigned thereunto duly authorized.



                               BROOKTROUT TECHNOLOGY, INC.
                               
                               
Date: November 12, 1997        By:
                                  -------------------------------------------- 
                                  Eric R. Giler
                                  President
                                  (Principal Executive Officer)
                               
                               
Date: November 12, 1997        By:
                                  -------------------------------------------- 
                                  Robert C. Leahy
                                  Vice President of Finance and Operations 
                                  and Treasurer
                                  (Principal Financial and Accounting Officer)





<PAGE>   14


                                  EXHIBIT INDEX


                                                                  Sequentially
Exhibit Number     Exhibit                                        Numbered Page

         11        Computation of Earnings Per Share

                   -For the three months ended                         15
                   September 30, 1997 and 1996

                   -For the nine months ended                          16
                   September 30, 1997 and 1996


         27        Financial Data Schedule                             17