A small amount of fine sediment will always make its way to the cup when brewing plunger coffee. This is because the plunger has a metal filter that allows fine particles to make it to the cup, resulting in a bolder brew with a heavy mouthfeel - one of the things people most love about plunger style coffee.
If you seem to have lots of large grounds in your cup, it might mean your grind is too fine - try coarsening up the grind.