In some cultures it's considered an insult not to accept their offer to eat with them.