Revert "Only show numbers for listed buffers"

This reverts commit 52654939e89532c2a125523274ea2958fbfaf983.