Abrupt aggression is often indicative of a disease process or a medical condition. In atypical cases like partial seizures within the limbic region, cats can demonstrate impulsive aggression that has no apparent trigger or environmental cause.
If your cat has tolerated handling prior and suddenly start displaying aggression while being stroked, schedule an appointment to see your vet.