The reason is that although these may well prove to be the brightest and darkest parts of your image, they are not necessary pure white or pure black and cannot therefore be used as reference colours. Instead you need to identify part of the image which you want to appear to be pure white or pure black, such as the white T-shirt of a person featured in the picture, a white window frame, etc. It is of course possible to have an image in which there are no reference whites or blacks, but this is quite unusual. For those images you will have to look for a reference grey, or colour correct by other means. In the following examples I am using Adobe PhotoShop.
Similar controls exist in other applications.
If you select the one coloured black and then click on part of the image, the curves are automatically adjusted so as to make the chosen colour black. The rest of the image's colours will shift accordingly. Likewise you can use the picker coloured white to select part of the image you want to be white. The task of identifying the brightest white part of an image can be made easier by adjusting the threshold of the image temporarily. By adjusting the threshold I can therefore see that the brightest white part of the image is the white of the window frames on the right side of the street.