"Going to pot" is a phrase that usually means someone is wasting their life.