They feared a loss in political and economic power if African Americans abandoned their one category