You should move to a finer grit once you've achieved an even, matte finish with no glossy spots. Each grit's purpose is to remove the scratches from the previous one, so inspect the surface carefully before progressing.