I told him that I've had four partners. I also feel bad about not telling him absolutely everything about my past, but I don't want to hurt him, or for him to get the wrong picture of me. In any case, and most importantly, it sounds like this little white lie is eating you up. If it's bothering you, then it's time to get it off your chest. However, you do have a point that he might not take it very well.
Some men get jealous and angry that their partner has been with other men.
In your case, though, you have already told him about your four relationships as compared to his one, and he's done fine with that knowledge. One day, in a quiet moment, start a conversation with your new boyfriend. Begin by telling him all the great things about your relationship, being very specific about what makes him special. Then move into the more difficult material. Be honest but spare him all the details.