And let us get up and go up to Bet El and I will make an alter there to the G-d who answers me in my day of trouble and was with me in the path which I took. The Midrash tells us not to entertain the thought that his sons or other wives were involved in idolatry but clearly many of those around him were indeed involved. An answer lies with the very nature of Avodah Zarah itself. It is easy to believe in a god who's sole job is to deal with one small task. Monotheism is much more complicated and requires that we believe that G-d alone governs everything in the universe.
Most idolatrous worship involves deities each of which has power over a certain area.
Primary categories, throughout the world, include powers over progeny and over the weather. It was very easy to fall into such beliefs because they give a sense of security, a false sense of security. It is much easier to do idolatry then Mitzvot of the Torah since all you have to do is believe and sometimes bring a sacrifice of some sort. Many people have trouble accepting the Pesukim which we say every day. You should serve him with all of your hearts and all your souls.