Only show numbers on listed buffers

This reverts commit c5a9e6d0290db4697084e35e1a4eacd1a09a26d1.