<PAGE>   1





                                                                      Exhibit 11
                                                                      ----------

                           BROOKTROUT TECHNOLOGY, INC.
<TABLE>

                     COMPUTATION OF INCOME PER COMMON SHARE
                (In thousands, except per share data, unaudited)

<CAPTION>
                                                  Three Months Ended
                                                     September 30,
                                                     -------------
                                                  1996          1995
                                                  ----          ----
<S>                                              <C>          <C>    
Primary Income Per Share:

Weighted average number of common
  and common equivalent shares outstanding:

    Common Stock                                  10,180        9,671

    Common equivalent shares
      resulting from options                         955          498
                                                 -------      -------

          Total                                   11,135       10,169
                                                 =======      =======

Net income                                       $ 2,218      $ 1,830
                                                 =======      =======

Net income per common share                      $  0.20      $  0.19
                                                 =======      =======

Fully Diluted Income Per Share:

Weighted average number of common
  and common equivalent shares
  outstanding

    Common stock                                  10,180        9,671

    Common equivalent shares
      resulting from options                       1,174          525
                                                 -------      -------

          Total                                   11,354       10,196
                                                 =======      =======

Net income                                       $ 2,218      $ 1,830
                                                 =======      =======

Net income per common share                      $  0.20      $  0.19
                                                 =======      =======

</TABLE>



<PAGE>   2


                                                                      Exhibit 11

                           BROOKTROUT TECHNOLOGY, INC.
<TABLE>

                   COMPUTATION OF INCOME PER COMMON SHARE (In
                  thousands, except per share data, unaudited)
<CAPTION>

                                                  Nine Months Ended
                                                     September 30,
                                                     -------------
                                                  1996         1995
                                                  ----         ----
<S>                                              <C>          <C>    
Primary Income Per Share:

Weighted average number of common
  and common equivalent shares outstanding:

    Common Stock                                   9,733        9,652

    Common equivalent shares
      resulting from options                         951          356
                                                 -------      -------

          Total                                   10,684       10,008
                                                 =======      =======

Net income                                       $ 4,078      $ 3,429
                                                 =======      =======

Net income per common share                      $  0.38      $  0.34
                                                 =======      =======

Fully Diluted Income Per Share:

Weighted average number of common
  and common equivalent shares
  outstanding

    Common stock                                   9,733        9,652

    Common equivalent shares
      resulting from options                       1,065          408
                                                 -------      -------

          Total                                   10,798       10,060
                                                 =======      =======

Net income                                       $ 4,078      $ 3,429
                                                 =======      =======

Net income per common share                      $  0.38      $  0.34
                                                 =======      =======

</TABLE>