A focus detection device with an imaging element, the imaging element including a plurality of imaging pixels (B, Gb, Gr, R) each with a light receiver and a plurality of focus detection pixels (208t, 208b, 2081, 208r) each with a light receiver including an opening position different from opening positions in the imaging pixels (B, Gb, Gr, R), the focus detection device comprising: a reading unit which is configured to control exposure of the imaging pixels (B, Gb, Gr, R) and the focus detection pixels (208t, 208b, 2081, 208r), and read pixel signals from the imaging pixels (B, Gb, Gr, R) or the focus detection pixels (208t, 208b, 2081, 208r) frame by frame; and a controller which is configured to perform a focus detection operation based on the pixel signals read from the focus detection pixels (208t, 208b, 2081, 208r); wherein the focus detection pixels (208t, 208b, 2081, 208r) comprise a first focus detection pixel (208t, 208b) in which the opening position is shifted in a first direction compared to the opening positions in the imaging pixels (B, Gb, Gr, R), and a second focus detection pixel (2081, 208r) in which the opening position is shifted in a second direction different from the first direction compared to the opening positions in the imaging pixels (B, Gb, Gr, R), characterized in that the focus detection device further comprises a reading range setting unit which is configured to set a reading range for the reading unit to read the pixel signals from the focus detection pixels (208t, 208b, 2081, 208r); the reading unit is configured to alternately start to read the pixel signals from the imaging pixels (B, Gb, Gr, R) and the pixel signals from the focus detection pixels (208t, 208b, 2081, 208r) in the reading range set by the reading range setting unit in synchronization with a synchronization signal that is input for each frame; the focus detection device further comprises a read pixel setting unit which is configured to set at least one of the first focus detection pixels (208t, 208b) and the second focus detection pixels (2081, 208r) for the reading unit to read the pixel signals; and the reading unit is configured to read the pixel signals based on the setting of the read pixel setting unit. a reading unit which is configured to control exposure of the imaging pixels (B, Gb, Gr, R) and the focus detection pixels (208t, 208b, 2081, 208r), and read pixel signals from the imaging pixels (B, Gb, Gr, R) or the focus detection pixels (208t, 208b, 2081, 208r) frame by frame; and a controller which is configured to perform a focus detection operation based on the pixel signals read from the focus detection pixels (208t, 208b, 2081, 208r); wherein the focus detection pixels (208t, 208b, 2081, 208r) comprise a first focus detection pixel (208t, 208b) in which the opening position is shifted in a first direction compared to the opening positions in the imaging pixels (B, Gb, Gr, R), and a second focus detection pixel (2081, 208r) in which the opening position is shifted in a second direction different from the first direction compared to the opening positions in the imaging pixels (B, Gb, Gr, R), characterized in that the focus detection device further comprises a reading range setting unit which is configured to set a reading range for the reading unit to read the pixel signals from the focus detection pixels (208t, 208b, 2081, 208r); the reading unit is configured to alternately start to read the pixel signals from the imaging pixels (B, Gb, Gr, R) and the pixel signals from the focus detection pixels (208t, 208b, 2081, 208r) in the reading range set by the reading range setting unit in synchronization with a synchronization signal that is input for each frame; the focus detection device further comprises a read pixel setting unit which is configured to set at least one of the first focus detection pixels (208t, 208b) and the second focus detection pixels (2081, 208r) for the reading unit to read the pixel signals; and the reading unit is configured to read the pixel signals based on the setting of the read pixel setting unit.