<PAGE>   1
HANOVER DIRECT, INC.                                                 EXHIBIT 11
COMPUTATION OF EARNINGS (LOSS) PER SHARE


<TABLE>
<CAPTION>
                                                                 (In 000's except per share amounts)

                                                                            13 WEEKS ENDED

                                                                 SEPTEMBER 28,         SEPTEMBER 27,
                                                                     1996                  1997
                                                                     ----                  ----
                                                                                      
<S>                                                               <C>                   <C>        
Net income (loss) before extraordinary item                       $   (28,431)          $   (3,421)
Extraordinary item                                                     (1,134)                --
                                                                  -----------           ----------
Net income (loss)                                                     (29,565)              (3,421)
Preferred stock dividends                                                 (59)                 (47)
                                                                  -----------           ----------
Net income (loss) applicable to common shareholders               $   (29,624)          $   (3,468)
                                                                  ===========           ==========
                                                                                      
Average shares of common stock outstanding during the period          114,252              200,330
                                                                                      
Total shares used to calculate PEPS*                                  114,252              200,330
                                                                                      
Primary earnings per share:                                                           
Earnings (loss) before extraordinary item                         $     (0.25)          $    (0.02)
Extraordinary item                                                      (0.01)                --
                                                                  -----------           ----------
Primary earnings per share                                              (0.26)               (0.02)
                                                                  ===========           ==========
                                                                                      
Average shares of common stock outstanding during the period          114,252              200,330
Total shares used to calculate FDEPS*                                 114,252              200,330
                                                                                      
Fully diluted earnings per share:                                                     
Earnings (loss) before extraordinary item                         $     (0.25)          $    (0.02)
Extraordinary item                                                      (0.01)                --
                                                                  -----------           ----------
Fully diluted earnings per share                                  $     (0.26)               (0.02)
                                                                  ===========           ==========
                                                                                      
Average shares of common stock outstanding during the period          114,252              200,300
                                                                                      
Basic earnings per share:                                                             
Earnings (loss) before extraordinary item                         $     (0.25)          $    (0.02)
Extraordinary item                                                      (0.01)                --
                                                                  -----------           ----------
Basic earnings per share                                          $     (0.26)          $    (0.02)
                                                                  ===========           ==========
</TABLE>                                                                     




*Per APB 15, when a net loss is reported, exercise or conversion of stock
options is not to be assumed.


                                       27