You can't do anything but reassign them again. Also, a greater number of unassigned settlers may lead to no new people showing up at your settlement.