The poet says so to hint at the fact that it does not matter what one had. People judge one's worth with money. If the poet does not have any, he will face indignation like others.