<PAGE>   1

                                                                      Exhibit 11

                           BROOKTROUT TECHNOLOGY, INC.

                     COMPUTATION OF INCOME PER COMMON SHARE
                (In thousands, except per share data, unaudited)

<TABLE>
<CAPTION>

                                                                   Three Months Ended
                                                                      September 30,
                                                                   1997            1996
                                                                   ----            ----
<S>                                                               <C>              <C>    

Primary Income Per Share:

Weighted average number of common and
  common equivalent shares outstanding

    Common Stock ............................................     10,709           10,180 
                                                                                          
    Common equivalent shares resulting from options .........        521              955 
                                                                 -------          ------- 
                                                                                          
          Total .............................................     11,230           11,135 
                                                                 =======          ======= 

Net income ..................................................    $   816          $ 2,218 
                                                                 =======          ======= 
                                                                                          
Net income per common share .................................    $  0.07          $  0.20 
                                                                 =======          ======= 
                                                                                          
Fully Diluted Income Per Share:                                                           
                                                                                          
Weighted average number of common and                                                     
  common equivalent shares outstanding                                                    
                                                                                          
    Common stock ............................................     10,709           10,180 
                                                                                          
    Common equivalent shares resulting from options .........        714            1,174 
                                                                 -------          ------- 
                                                                                          
          Total .............................................     11,423           11,354 
                                                                 =======          ======= 
                                                                                          
Net income ..................................................    $   816          $ 2,218 
                                                                 =======          ======= 
                                                                                          
Net income per common share .................................    $  0.07          $  0.20 
                                                                 =======          ======= 
                                                                                       
</TABLE>
                                                                 




<PAGE>   2
                                                                      Exhibit 11

                           BROOKTROUT TECHNOLOGY, INC.

                     COMPUTATION OF INCOME PER COMMON SHARE
                (In thousands, except per share data, unaudited)


<TABLE>
<CAPTION>
                                                                   Nine Months Ended
                                                                      September 30,
                                                                   1997          1996
                                                                 -------        ------- 
<S>                                                               <C>             <C>    

Primary Income Per Share:

Weighted average number of common and
  common equivalent shares outstanding

    Common Stock ..............................................   10,698          9,733  
                                                                                        
    Common equivalent shares resulting from options ...........      596            951 
                                                                 -------        ------- 
                                                                                        
          Total ...............................................   11,294         10,684 
                                                                 =======        ======= 
                                                                                        
Net income ....................................................  $   902        $ 4,078 
                                                                 =======        ======= 
                                                                                        
Net income per common share ...................................  $  0.08        $  0.38 
                                                                 =======        ======= 
                                                                                        
Fully Diluted Income Per Share:                                                         
                                                                                        
Weighted average number of common and                                                   
  common equivalent shares outstanding                                                  
                                                                                        
    Common stock ..............................................   10,698          9,733 
                                                                                        
    Common equivalent shares resulting from options ...........      614          1,065 
                                                                 -------        ------- 
                                                                                        
          Total ...............................................   11,312         10,798 
                                                                 =======        ======= 
                                                                                        
Net income ....................................................  $   902        $ 4,078 
                                                                 =======        ======= 

Net income per common share ...................................  $  0.08        $  0.38 
                                                                 =======        ======= 
</TABLE>