The literal reason given in the poem is that the man simply has no time because he's too busy working. This is another example of the poem subverting the grand or holy for the mundane. Instead of becoming a prophet to spread their message, they would rather quietly and solitarily grow tomatoes.