Many Muslims criticized the Umayyads for having too many non-Muslim, former Roman administrators in their government