In buying spectacles the needless outlay for the right lens soon reduced him to poverty , and the Man to Whom Time Was Money had to sustain life by fishing from the end of a wharf .