Now that that's gone, you are missing even more than the woman you were having the affair with. You are missing your focus, the excitement in your life, the secret you lived for. You looked forward to it and thought about it afterwards. Now you have nothing to look forward to and nothing to do but dwell on what you had that you don't have anymore. Interestingly, men suffer more when a relationship is over than do women.
Women have their friends and talk it out endlessly using their female support system.
Men feel they have to go it alone, especially in a case like yours where you really can't talk to anyone about your pain. You're right, there aren't a lot of pity points you can rack up for messing with another man's wife for six years, but you are in pain and that's what counts. Eventually you'll find someone new. But unless you take steps to rebuild your life and replace your lost love, you could continue to obsess and suffer for a long time. Now that you are ready to accept that you will never have this woman and to give up hope of getting her back, you are ready to move on.