The point is that overall impact goes up with population, but it also goes up with level of affluence, i.e. the richer we get the more we consume per capita. Also impact depends on technology, although it is unclear exactly how. Sometimes technology makes impact go up, and sometimes it make it go down, e.g. motorized pleasure craft, vs, a more efficient solar pump. The hope is that as technology advances, we will find better ways to do things which have less impact per capita. Well, in one sense it's not, because even if we had 10 times today's population, if there were no CFCs escaping into the air then we wouldn't have a problem.
Humans are imperfect.
No matter how hard we as a race try, it seems we keep screwing up, with CFCs, with DDT, etc. But even those of us who want to do the right thing to protect the environment can't, because we too have to have jobs to eat, and thus ultimately generate waste, some fraction of which will undoubtedly be toxic. So OP is really a question more than an answer. But P and A seem to be the trouble spots. Some people say we should have as large a P as we want, if we just reduce our A, or in other words, large families are ok if we are willing to reduce our quality of life , e.g. less health care, less personal transportation, lower home heat comfort levels, etc.