An imaging element comprising: a plurality of pixels each having an incident angle directivity for incident light incident without passing through both of an imaging lens and a pinhole, said incident angle sensitivity being defined by a portion of the pixel which is covered by a light shielding film wherein the plurality of pixels includes a plurality of multidirectional pixels arranged in a matrix, the multidirectional pixels having incident angle directivity both in a row direction and in a column direction of the matrix, wherein each multidirectional pixel in a row has the same incident angle directivity in the column direction and each multidirectional pixel in a column has the same incident angle directivity in the row direction, and the incident angle directivity changes from one row to the next and from one column to the next; and a first plurality of unidirectional pixels having incident angle directivity only in the row direction of the matrix, each unidirectional pixel of said first plurality corresponding to a column of the matrix and having the same incident angle directivity as the multidirectional pixels in said column; a second plurality of unidirectional pixels having the incident angle directivity only in the column direction of the matrix, each unidirectional pixel of said second plurality corresponding to a row of the matrix and having the same incident angle directivity as the multidirectional pixels in said row; and; a generation unit configured to generate a restoration matrix to be used when generating a restored image from a detection image obtained by the multidirectional pixels, using calibration information that is information regarding calibration of the incident angle directivity of the multidirectional pixels generated using the first plurality of unidirectional pixels and the second plurality of unidirectional pixels. a plurality of pixels each having an incident angle directivity for incident light incident without passing through both of an imaging lens and a pinhole, said incident angle sensitivity being defined by a portion of the pixel which is covered by a light shielding film wherein the plurality of pixels includes a plurality of multidirectional pixels arranged in a matrix, the multidirectional pixels having incident angle directivity both in a row direction and in a column direction of the matrix, wherein each multidirectional pixel in a row has the same incident angle directivity in the column direction and each multidirectional pixel in a column has the same incident angle directivity in the row direction, and the incident angle directivity changes from one row to the next and from one column to the next; and a first plurality of unidirectional pixels having incident angle directivity only in the row direction of the matrix, each unidirectional pixel of said first plurality corresponding to a column of the matrix and having the same incident angle directivity as the multidirectional pixels in said column; a plurality of multidirectional pixels arranged in a matrix, the multidirectional pixels having incident angle directivity both in a row direction and in a column direction of the matrix, wherein each multidirectional pixel in a row has the same incident angle directivity in the column direction and each multidirectional pixel in a column has the same incident angle directivity in the row direction, and the incident angle directivity changes from one row to the next and from one column to the next; and a first plurality of unidirectional pixels having incident angle directivity only in the row direction of the matrix, each unidirectional pixel of said first plurality corresponding to a column of the matrix and having the same incident angle directivity as the multidirectional pixels in said column; a second plurality of unidirectional pixels having the incident angle directivity only in the column direction of the matrix, each unidirectional pixel of said second plurality corresponding to a row of the matrix and having the same incident angle directivity as the multidirectional pixels in said row; and; a generation unit configured to generate a restoration matrix to be used when generating a restored image from a detection image obtained by the multidirectional pixels, using calibration information that is information regarding calibration of the incident angle directivity of the multidirectional pixels generated using the first plurality of unidirectional pixels and the second plurality of unidirectional pixels.