He has just healed the man at the pool of Bethesda on the Sabbath. Then he is picking grain on the Sabbath on the way through a field. According to the OT, it was OK to pick some grain as you passed through someone's field. What the Pharisees were concerned with was that He was doing it on the Sabbath. David ate the shewbread in the temple, and that was OK, because the rule not to eat the shewbread was not more important than the starvation of humans.
Priests work hardest on Sabbath.
God does not want ritual service without the heart. And the Sabbath was made for the benefit of man and not vice-versa. The Pharisees made the Sabbath more important than humanity. Humans were serving the Sabbath. It was a burden to them.