What happened to the number of free colored people for the first few decades after the war