But he has had to backtrack from simply raising the detention limit in the face of mounting opposition not just from political opponents and civil rights groups , but from senior legal figures and within his own Labour Party .