Hunger should only go down if you broke the block. Not during the whole breaking process.