The key to relationship is communication, and that's what makes trust develop with time. Also the behaviour of the partner, and the sharing are the main resaons of developing trust. Your fears are certainly not abnormal or wrong, they are natural and would have happened to most people, at the same time past alone cannot be the reason to judge a present relationship. You need to increase communication, and sharing to ensure that you have a better perspective on this matter. As for the family related statement, address this with him, and talk about your views, and see his response regarding the same. Communication as I said is the key.