A signal processing device comprising: a restoration unit configured to restore one restored image by using a plurality of detection signal sets each obtained by one of a plurality of imaging elements in one of a plurality of states in which at least one of a position or orientation of the imaging element with respect to a subject is different, wherein the imaging elements each include a plurality of pixel output units configured to receive incident light from the subject incident without an intervention of an imaging lens or a pinhole and to each output one detection signal indicating an output pixel value modulated by an incident angle of the incident light, and wherein the imaging elements are each configured to output a detection signal set including a plurality of detection signals output from the plurality of pixel output units, wherein the restoration unit is configured to restore the one restored image by using the plurality of detection signal sets and a coefficient set group obtained by the restoration unit and corresponding to a distance between each of the plurality of imaging elements and a surface of the subject, wherein the coefficient set group comprises a plurality of coefficients obtained, for each of the pixel output units, on the basis of an incident angle of a light beam from a point light source on the surface of the subject to that pixel output unit, a weight Wx corresponding to the modulation of the detection signal by the incident angle of the light beam from the point light source in a horizontal direction of that pixel output unit and a weight Wy corresponding to the modulation of the detection signal by the incident angle of the light beam from the point light source in a vertical direction of that pixel output unit. a restoration unit configured to restore one restored image by using a plurality of detection signal sets each obtained by one of a plurality of imaging elements in one of a plurality of states in which at least one of a position or orientation of the imaging element with respect to a subject is different, wherein the imaging elements each include a plurality of pixel output units configured to receive incident light from the subject incident without an intervention of an imaging lens or a pinhole and to each output one detection signal indicating an output pixel value modulated by an incident angle of the incident light, and wherein the imaging elements are each configured to output a detection signal set including a plurality of detection signals output from the plurality of pixel output units, wherein the restoration unit is configured to restore the one restored image by using the plurality of detection signal sets and a coefficient set group obtained by the restoration unit and corresponding to a distance between each of the plurality of imaging elements and a surface of the subject, wherein the coefficient set group comprises a plurality of coefficients obtained, for each of the pixel output units, on the basis of an incident angle of a light beam from a point light source on the surface of the subject to that pixel output unit, a weight Wx corresponding to the modulation of the detection signal by the incident angle of the light beam from the point light source in a horizontal direction of that pixel output unit and a weight Wy corresponding to the modulation of the detection signal by the incident angle of the light beam from the point light source in a vertical direction of that pixel output unit.