The main part of the issue with climate change is its relationship to the state of human civilization.
What the climate did a million years ago doesn't really matter, because humans (as such) didn't exist then.