Notice he did not say that physically hearing the message was truly hearing. You see, you can hear and not hear. Ask any wife whether that is true. Faith only comes when we have truly heard with our hearts. Obviously, it was possible, even probable, that many would not have ears to hear, even though they had physically and intellectually heard the message.
True hearing is revealed in the response to the message.
We know that a person has truly heard when that individual responds in faith. Faith comes by hearing, hearing with the heart. Sadly, many will not hear with the heart. They cannot exercise faith because they have no spiritual ears. But God never gives up declaring the message to all.