Disable linum from the buffer menu and the speedbar.