While sandpaper can be used, it is generally recommended to use steel wool or a wire brush for removing rust from brake rotors. They are specifically designed for this purpose and provide better results without causing damage to the rotor surface.