An overlay measurement apparatus comprising: a polarized light source configured to illuminate a substrate with a polarized light beam; an optical system configured to capture light that is scattered by the sample caused by the polarized light beam, the optical system comprising, a first non-polarizing beam splitter configured to receive the polarized light beam from the polarized light source, direct the light to the substrate via a projection system, and receive a reflected light beam from the substrate via the projection system, the reflected light beam comprising a zeroth order diffraction component; a second non-polarizing beam splitter configured to receive a transmitted light beam from the first non-polarizing beam splitter, wherein the beam splitter axes of the first and second non-polarizing beam splitters are mutually perpendicular; a crossed polarizer configured to receive the split light beams from the second non-polarizing beam splitter and to transmit an in-plane image and crossed polarized image to a detector configured to detect the crossed polarized image and to produce a signal therefrom, and a processing unit configured to receive the signal from the detector and to process the crossed polarized image for overlay metrology measurement using asymmetry data derived from the crossed polarized image, characterized in that the overlay measurement apparatus is configured to base an overlay measurement only on the asymmetry data derived from the crossed polarized image. a polarized light source configured to illuminate a substrate with a polarized light beam; an optical system configured to capture light that is scattered by the sample caused by the polarized light beam, the optical system comprising, a first non-polarizing beam splitter configured to receive the polarized light beam from the polarized light source, direct the light to the substrate via a projection system, and receive a reflected light beam from the substrate via the projection system, the reflected light beam comprising a zeroth order diffraction component; a second non-polarizing beam splitter configured to receive a transmitted light beam from the first non-polarizing beam splitter, wherein the beam splitter axes of the first and second non-polarizing beam splitters are mutually perpendicular; a crossed polarizer configured to receive the split light beams from the second non-polarizing beam splitter and to transmit an in-plane image and crossed polarized image to a detector configured to detect the crossed polarized image and to produce a signal therefrom, and a processing unit configured to receive the signal from the detector and to process the crossed polarized image for overlay metrology measurement using asymmetry data derived from the crossed polarized image, a first non-polarizing beam splitter configured to receive the polarized light beam from the polarized light source, direct the light to the substrate via a projection system, and receive a reflected light beam from the substrate via the projection system, the reflected light beam comprising a zeroth order diffraction component; a second non-polarizing beam splitter configured to receive a transmitted light beam from the first non-polarizing beam splitter, wherein the beam splitter axes of the first and second non-polarizing beam splitters are mutually perpendicular; a crossed polarizer configured to receive the split light beams from the second non-polarizing beam splitter and to transmit an in-plane image and crossed polarized image to a detector configured to detect the crossed polarized image and to produce a signal therefrom, and a processing unit configured to receive the signal from the detector and to process the crossed polarized image for overlay metrology measurement using asymmetry data derived from the crossed polarized image, characterized in that the overlay measurement apparatus is configured to base an overlay measurement only on the asymmetry data derived from the crossed polarized image.