Nothing is too good for the Kohen. For the ani, poor man, in contrast, we leave over the corner of the field upon which all of the workers have probably already tread. We give him undeveloped twigs and fallen fruit. Horav Zalmen Sorotzkin, zl, feels that the Torah's distribution process for the various contributions is practical, given the fact that we must take into account the human nature of the donor. Thus, he gives the finest and best of his crops.
People view the poor and destitute through an entirely different prism.
The poor man has had his chance to earn a living. He received a portion in Eretz Yisrael, just like everybody else. Pure laziness must be motivating him to beg rather than to work. Hashem knows, of course, that poverty and wealth are Divine decrees. Due to reasons beyond our limited scope of perception, Hashem gives the poor man's livelihood to the rich man.