Phidias fell out of political favor due to his association with prominent figures, including General Pericles. He was accused of stealing gold and ivory during the creation of the Athena Parthenos statue, which led to his imprisonment and eventual fate.