A processor implemented method of image processing for impurity identification, comprising: obtaining, via one or more hardware processors, an RGB-polarization P, image, an RGB image, and RGB-D image, of a scene, as inputs, wherein the RGB-P image is captured when an unpolarized light from a controlled light source falls on an object of interest, and reflected light, which becomes polarized is captured using a polarization camera from multiple polarization filter angles and estimate Stokes vector parameters with at least three sets of intensities corresponding to three polarization filter angles; pre-processing the obtained RGB-P image, RGB and RGB-D images to remove motion blur, via the one or more hardware processors; generating a Degree of Polarization, DoP, map from at least one plane of the RGB-P image, via the one or more hardware processors; constructing a first graph, a second graph and a third graph, via the one or more hardware processors, wherein the first graph is constructed for the DoP map, the second graph is constructed for the RGB image, and the third graph is constructed for the RGB-D image; constructing a joint graph spectrum, via the one or more hardware processors, by merging the first graph, the second graph and the third graph, wherein the joint graph spectrum is closer to a spectrum of the first graph and smooth with respect to the second graph and the third graph, wherein closer refers to extent of overlap between the joint graph spectrum and the spectrum of the first graph, wherein the smoothness is ensured by minimizing graph total variation, GTV, of the second graph and the third graph; performing spectral clustering using the joint graph spectrum, comprising: identifying a plurality of segments present in the joint graph spectrum; and mapping the plurality of segments with the DoP map results in identifying values from the DoP map that match each of the plurality of segments and assigning the identified values of DoP map to each of the plurality of segments; and determining, based on size of each of the plurality of segments and mapped value of each of the plurality of segments, presence or absence of one or more impurity materials in the scene, via the one or more hardware processors. obtaining, via one or more hardware processors, an RGB-polarization P, image, an RGB image, and RGB-D image, of a scene, as inputs, wherein the RGB-P image is captured when an unpolarized light from a controlled light source falls on an object of interest, and reflected light, which becomes polarized is captured using a polarization camera from multiple polarization filter angles and estimate Stokes vector parameters with at least three sets of intensities corresponding to three polarization filter angles; pre-processing the obtained RGB-P image, RGB and RGB-D images to remove motion blur, via the one or more hardware processors; generating a Degree of Polarization, DoP, map from at least one plane of the RGB-P image, via the one or more hardware processors; constructing a first graph, a second graph and a third graph, via the one or more hardware processors, wherein the first graph is constructed for the DoP map, the second graph is constructed for the RGB image, and the third graph is constructed for the RGB-D image; constructing a joint graph spectrum, via the one or more hardware processors, by merging the first graph, the second graph and the third graph, wherein the joint graph spectrum is closer to a spectrum of the first graph and smooth with respect to the second graph and the third graph, wherein closer refers to extent of overlap between the joint graph spectrum and the spectrum of the first graph, wherein the smoothness is ensured by minimizing graph total variation, GTV, of the second graph and the third graph; performing spectral clustering using the joint graph spectrum, comprising: identifying a plurality of segments present in the joint graph spectrum; and mapping the plurality of segments with the DoP map results in identifying values from the DoP map that match each of the plurality of segments and assigning the identified values of DoP map to each of the plurality of segments; and identifying a plurality of segments present in the joint graph spectrum; and mapping the plurality of segments with the DoP map results in identifying values from the DoP map that match each of the plurality of segments and assigning the identified values of DoP map to each of the plurality of segments; and determining, based on size of each of the plurality of segments and mapped value of each of the plurality of segments, presence or absence of one or more impurity materials in the scene, via the one or more hardware processors.