I'm just wondering if I should start the divorce and go. In your husband's case, this may mean his understanding that he is going to lose his marriage if he doesn't admit that he has a drinking problem and does something about it. As a recovering alcoholic and a psychiatrist, I wholly support and recommend both AA and Al-anon, the AA inspired support group that helps significant others of alcoholics. You will learn more from them than anything I can tell you. Even before you confront your husband about his drinking, I would recommend that you attend several Al-anon meetings.
Good lovers are easier to find than good marriages.
Give your marriage a chance to 'lose the weight' it has accumulated before you walk away. The answer to this question may tell you if you want to work on repairing your relationship with your husband, or file for divorce. The pleasurable feelings you enjoyed during the affair may be related to a contradiction in values between your husband's culture and yours. If you and your husband are committed to repairing your relationship, you should attend counseling sessions together. That's why you married him.