High Dynamic Range Astronomical Imaging

High dynamic range imaging methods that combine multiple exposures involve a number of processing details, such as frame alignment, motion artifact removal, and pixel data weighting. When applied to astronomical scenes, these steps require additional consideration. Specifically, motion blur artifacts must be corrected by techniques that are tolerant of saturated pixels, successive image frames need to be registered with point source precision, and the weighting of shorter exposures in the HDR compositing step are limited by signal-tonoise considerations. In addition, if there is light contributed by the foreground sky (common in light polluted regions), it needs to be offset and balanced in a blackpoint normalizing procedure. This paper identifies and describes these HDR issues specific to creating astrophotographic imagery. Introduction High dynamic range (HDR) imaging has become a popular tool among photographers as a result of pioneering work in computational photography [2, 9], and the introduction of software tools and utilities to assemble and manage HDR images. It is particularly effective in outdoor pictures where the scene radiances cover a range that exceeds what can be captured in any single film or digital exposure. The basic technique is to take a set of exposures, bracketing the range so that everything in the scene is properly exposed on at least one frame. The frames are then computationally blended to obtain a high dynamic range image. A tone mapping operation is performed to bring out the desired details for a given display or print. Sophisticated tone mappings take into account the spatial variations of intensity across the image, mimicking the adaptation of the human eye to a high dynamic range scene. It would seem that HDR methods would be a natural fit for astrophotography, where the objects in a scene cover an enormous range of brightness, from the sun, moon and planets to faint nebulas and galaxies. There are a number of difficulties however, some of which are intrinsic to the HDR process but are exacerbated by astronomical subject matter (frame alignment, exposure weighting, motion artifacts), and others that are unique to it (sensor noise, sky fog). 1. Alignment and registration Registration is a prerequisite for blending the individual exposures to form an HDR image. The frames are aligned so that the pixels are correlated to the same surface points of the same objects in the scene. Accurate registration yields good sharpness and preserves texture of course, but the objects that dominate astrophotos are stars; point sources that, if not tightly aligned from frame to frame, will cause the HDR merging process to fail. Most terrestrial scenes can be adequately aligned by affine transformations of scaling and rotation, but image formation by lenses is not perfectly rectilinear, and this is only a first step toward the registration required here. Astrophotographic targets may not change their relative positions in the sky, but pictures of them can be acquired on different nights or even different years, making it unlikely that their positions within the frame will be identical. This is understood in the astronomical imaging community, where it is common to add images together to increase the signal to noise ratio, and the software tools to help with this summation usually includes provisions for alignment. Not all perform the local warping necessary for the tight registration we need, but one such tool that does is a program called RegiStar [8]. It identifies the stars in a scene and performs the operations necessary to register additional frames to match the individual stars in the reference frame. 2. Motion deblur Astrophotos are taken at the parametric edge of conventional photography. The exposures can be very long, seconds, minutes, even hours, and the optics can be operating at high magnification; focal lengths measured in meters is not uncommon. This makes the exposures vulnerable to motion errors, from vibrations induced by the shutter, (or by wind on the equipment), and from tracking errors in keeping perfect aim at the targets as the earth moves slowly beneath them. There have been a variety of methods developed to remove motion blur from photographs [1, 3, 4, 5, 10]. One of the popular techniques in astronomy is Lucy-Richardson deconvolution, an iterative method to estimate what a source image must have been in order to produce the distorted image observed. Since each image of a star is a representation of the point spread function of the system (including the motion), it can be used in the deconvolution algorithm. However, convolution and deconvolution are linear operators that assume the observed pixel values are the sum of the motion-disturbed signals. This is true for all of the nonsaturated pixels in a calibrated radiance or luminance image, but not for those that have been clipped. The result is that artifacts occur in and near these regions. In most applications, the saturated regions are not of interest, but for our purpose, we care deeply about them. The saturated pixel locations matter because we will be identifying them to be rejected from one frame, and supplied by another. They must not artificially become suddenly unsaturated, nor should they cause other nearby pixels to become incorrectly saturated. In the deconvolution process, the saturated pixels represent limited information about the intensities in the scene. We must not ignore them, but rather use them to provide bounds on the amplitudes of the other pixels in the neighborhood. To this end, a saturation-preserving version of the Lucy-Richardson deconvolution algorithm was developed. It permitted use of frames that would otherwise have been highly misrepresentative in the HDR stack. Some examples of its behavior are shown. Of course the best answer is to avoid the motion artifacts to begin with. Camera mirror lockup, solid mechanical mounting, guided tracking, are methods that can reduce the blur. Although we have discussed it second, it should be noted that numerical deblurring methods if used, should be performed before frame registration. 30 sec 4 sec 1/30 sec, (2x spatial scale) Figure 1. Some examples showing the results of motion deblur algorithms. The top row are selected samples of motion blurred exposures in an HDR sequence. The middle row shows the application of Lucy-Richardson deconvolution supplied in the Matlab Imaging Toolbox. The bottom row is output from DeblurLucySat, a Matlab script developed to account for pixel saturation. 3. Noise In astrophotography, it’s all about the noise. Every effort is made to minimize it: thermally cooled sensors, dark frame removal techniques, long integration times and multiple exposures are all put to bear on the problem. Depending on the subject and the imaging conditions, different noise sources dominate. In some situations detector noise is the problem. This is the case when the sky is very dark compared to the scene, a condition that is becoming increasingly compromised by light pollution. When the sky is not dark, this background level overwhelms the sensor thermal and fixed pattern noise (in a good sensor, including those in modern digital SLRs). Each case (dark sky, bright sky) requires slightly different treatment when blending frames to make an HDR image. Conventionally, when individual frames are blended to form an HDR image, the near-black and the near-white pixels are rejected, as their information will be provided by some other frame in the set. The others are accepted, scaled by the exposure factor, and given a weighting that indicates its importance compared to corresponding pixels from other frames that might also contribute to that image location. A variety of weighting functions have been proposed over the development of HDR [6, 9, 11], and the subject will be discussed further below, but for now we note that we must not discard the near-black pixels too easily, since they contain much if not most, of the information we seek in an astronomic scene. The importance we give the near-black pixels is a delicate balance between signal level and noise level. Because the pixel value is scaled by the exposure factor (and this factor can be very large), it is critical to avoid scaling the noise rather than the signal. 3.1. Sensor noise There are a number of noise sources associated with image sensors, but one of the important ones is dark current, the signal generated when no light is present. It is strongly temperature dependent. There are others as well, amplifier noise, read noise, hot and cold pixels, fixed pattern noise, cosmic ray hits, and other noise sources that may be specific to a sensor’s technology or its integration in an imaging system. A section obtained from a frame taken with a Canon EOS20Da is shown, highly amplified (figure 2). The noise in the Canon sensor is remarkably low, but we want to find its distribution so that we can estimate the likelihood that a given pixel amplitude represents signal rather than noise. If we look at the histogram, we can see that the bulk of the noise appears to be Gaussian with a small mean and sigma, but there is a residual non-gaussian “tail” that extends to much larger values. Figure 2. a) A 320x256 section from the background of a 1-sec exposure made by a Canon EOS-20Da at f/9.9, ISO-800. The values have been multiplied by 255 from their 16-bit linear representation to show the visual character of the noise. b) To see the behavior in detail, a log of the histogram is taken. It is obvious that the fitted normal distribution falls off too rapidly compared with the observed noise. c) A normal Gaussian (solid) can be fit to the early part of the distribution function, but the tail is best fit by a Weibull model (dashed). A blend of the two noise processes is needed to match the data. The exponential falloff can be modeled as a Weibull distribution [7], and blended with the Gaussian that matches the d