Two important reasons. First, the public schools are too big. The second problem is lack of order or discipline, which is contributed to by the bigness. Chaos is the rule, and chaos tends to be destructive of the educational process. For them the non-government schools they are attending are more integrated than the public schools.
Charter schools are, after all, government schools.
However well they may start out, it is easy for the charter schools to relapse into being typical government schools. The readers can go invade the schools. When I was in grade school, my mother came to school to visit. That was the public school, and my mother was there in the back of the room. It was unusual, but it did happen.