If you look away while listening, it shows lack of interest and a short attention span. If you fail to maintain eye contact while speaking, at a minimum it shows lack of confidence in what you are saying and at worst may send the subtle message that you are lying. Do not just assume you have good eye contact. Ask others if you ever lack proper eye contact. If they respond that they did notice, ask if it was during speaking or listening.
Some people maintain excellent eye contact while listening, but lose eye contact when speaking.
Or vice versa. Next, watch yourself on videotape. Then sit down with a friend and practice until you are comfortable maintaining sincere, continuous eye contact. It is almost as if four years of college has left some students brain dead or worse. Take a good, long, hard look at yourself in the mirror.