<PAGE>   1

                       SECURITIES AND EXCHANGE COMMISSION
                             Washington, D.C. 20549

                            -------------------------
                                   FORM 10-K/A

        [X]  ANNUAL REPORT PURSUANT
             TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934

                   For the fiscal year ended December 31, 1998

        [ ]  TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES
             EXCHANGE ACT OF 1934

                     For the transition period from __ to__.

                           Commission File No. 0-20698

                           BROOKTROUT TECHNOLOGY, INC.
             (Exact name of registrant as specified in its charter)

                 MASSACHUSETTS                            04-2184792
                (State or other                        (I.R.S. Employer
        incorporation or organization)                Identification No.)

                 410 FIRST AVENUE, NEEDHAM, MASSACHUSETTS 02494
               (Address of principal executive offices)(Zip Code)

       Registrant's telephone number, including area code: (781) 449-4100

        Securities registered pursuant to Section 12(b) of the Act: None

    Securities registered pursuant to Section 12(g) of the Act: Common Stock

        Indicate by check mark whether the registrant (1) has filed all reports
required to be filed by Section 13 or 15(d) of the Securities Exchange Act of
1934 during the preceding 12 months (or for such shorter period that the
registrant was required to file such reports), and (2) has been subject to such
filing requirements for the past 90 days. Yes [X] No [ ]

        Indicate by check mark if disclosure of delinquent filers pursuant to
Item 405 of Regulation S-K is not contained herein, and will not be contained,
to the best of Registrant's knowledge, in definitive proxy or information
statements incorporated by reference in Part III of this Form 10-K or any
amendment to this Form 10-K. [X]

As of March 24, 1999, the aggregate market value of the voting stock held by
non-affiliates of the registrant was approximately $112.6 million, based on the
closing price on such date of the Company's Common Stock on the Nasdaq
("Nasdaq") National Market. As of March 24, 1999, 10,907,012 shares of Common
Stock, $.01 par value per share, were outstanding.

                       DOCUMENTS INCORPORATED BY REFERENCE

        Portions of (1) the Company's Annual Report for the fiscal year ended
December 31, 1998 are incorporated into Part II of this Form 10-K and (2) the
Company's Proxy Statement relating to the 1999 Annual Meeting of Shareholders of
the Company are incorporated into Part III of this Form 10-K.




<PAGE>   2




                                     PART IV

ITEM 14.    EXHIBITS, FINANCIAL STATEMENT SCHEDULES AND REPORTS ON FORM 8-K

        (a) (3)  Exhibits

        The following exhibits are filed as part of this report on Form 10-K/A.
Such filings are made by incorporation by reference to the Company's Current
Report on Form 8-K dated September 9, 1998, as filed on September 14, 1998.

Exhibit
   No.                                            Title
-------                                           -----

   3.1                        Certificate of Vote of Directors Establishing a
                              Series of a Class of Stock of the Company
                              classifying and designating the Series A Junior
                              Participating Cumulative Preferred Stock

   4.1                        Shareholder Rights Agreement, dated as of
                              September 9, 1998, between the Company and State
                              Street Bank and Trust Company, as Rights Agent




                                        2


<PAGE>   3




                                   SIGNATURES

   Pursuant to the requirements of Section 13 or 15(d) of the Securities
Exchange Act of 1934, as amended, the Registrant has duly caused this report to
be signed on its behalf by the undersigned, thereunto duly authorized.

                                                 BROOKTROUT TECHNOLOGY, INC.


                                                 By: /s/ Eric R. Giler
                                                     ---------------------------
                                                     Eric R. Giler
                                                     President

Date: May 13, 1999


                                   SIGNATURES

        Pursuant to the requirements of the Securities Exchange Act of 1934,
this report has been signed below by the following persons on behalf of the
registrant in the capacities and on the dates indicated.

<TABLE>
<CAPTION>
             Signatures                                Title                                   Date
             ----------                                -----                                   ----

<S>                                            <C>                                         <C> 
/s/ Eric R. Giler                              President and Director (Principal           May 13, 1999
--------------------------------------         Executive Officer)
Eric R. Giler                                  

/s/ Robert C. Leahy                            Vice President of Finance                   May 13, 1999
--------------------------------------         and Operations and Treasurer
Robert C. Leahy                                (Principal Financial and
                                               Accounting Officer)

/s/ David W. Duehren                           Vice President and Director                 May 13, 1999
--------------------------------------
David W. Duehren

/s/ Patrick T. Hynes                           Vice President and Director                 May 13, 1999
--------------------------------------
Patrick T. Hynes

/s/ Robert G. Barrett                          Director                                    May 13, 1999
--------------------------------------
Robert G. Barrett

/s/ David L. Chapman                           Director                                    May 13, 1999
--------------------------------------
David L. Chapman

/s/ W. Brooke Tunstall                         Director                                    May 13, 1999
--------------------------------------
W. Brooke Tunstall
</TABLE>



                                       3