Extreme imaging using cell phones
ICCV Extreme Imaging Workshop
December 17, 2015
(revised September 15, 2016)

Marc Levoy
Principal Engineer
Google Research

Professor, Emeritus
Computer Science Department
Stanford University

Really? Extreme imaging using cell phones?
✦

small aperture• limits the number of photons it can gather• noisy images in low light✦small sensor• limits the number of electrons each pixel can store• poor dynamic range✦

small lens module• limits the length of its optical path
• no telephoto, no zoom

2

© Marc Levoy

Jesse Levinson Canon 10D, 28mm f/4, 3 min, ISO 100, 4 image pano(video available on YouTube at https://youtu.be/S7lbnMd56Ys, but the app is not currently available to the public)

Canon 60D, f/5.6, ISO 12800, 1/8 second

Sony a7II, Leica 35mm prime lens, f/2, ISO 6400,1/8 second

SeeInTheDarkExample computational photography app:
HDR+ mode on Nexus phones

8© Marc Levoy

Typical approach to HDR✦

exposure bracketing• capture images with varying exposure• combine highlights from short exposure

with shadows from long exposure✦

hard to robustly handle camera shake or object motion• noise level differs between exposures• saturated areas cannot be aligned at all

9© Marc Levoy

HDR in the Google camera app[to be published in Proc. SIGGRAPH Asia 2016]✦

capture a burst of under-exposed images• same exposure on all images in burst• avoids blowing out highlights

10© Marc Levoy

Stanford Frankencamera architecture
and FCam API [Adams SIGGRAPH 2010]
Sensor sensor;
Flash flash;vector<Shot> burst(2);burst[0].exposure = 1/200.;
burst[1].exposure = 1/30.;
Flash::FireAction fire(&flash);fire.time = burst[0].exposure/2;
burst[0].actions.insert(fire);
sensor.stream(burst);

11

while (1) {Frame flashFrame =sensor.getFrame();Frame noflashFrame =sensor.getFrame();}© Marc Levoy

Android Camera HAL 3 architecture
and Camera2 API (Eddy Talvala and others)
Sensor sensor;
Flash flash;vector<Shot> burst(2);burst[0].exposure = 1/200.;
burst[1].exposure = 1/30.;
Flash::FireAction fire(&flash);fire.time = burst[0].exposure/2;
burst[0].actions.insert(fire);
sensor.stream(burst);

12

while (1) {Frame flashFrame =sensor.getFrame();Frame noflashFrame =sensor.getFrame();}© Marc Levoy

Android Camera HAL 3 architecture
and Camera2 API (Eddy Talvala and others)
Sensor sensor;
Flash flash;vector<Shot> burst(2);burst[0].exposure = 1/200.;
burst[1].exposure = 1/30.;
Flash::FireAction fire(&flash);fire.time = burst[0].exposure/2;
burst[0].actions.insert(fire);
sensor.stream(burst);

13

while (1) {Frame flashFrame =sensor.getFrame();Frame noflashFrame =sensor.getFrame();}© Marc Levoy

try {// This is how to tell the camera to trigger.
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER,
CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER_START);// Tell #mCaptureCallback to wait for the precapture sequence to be set.
mState = STATE_WAITING_PRECAPTURE;
mCaptureSession.capture(mPreviewRequestBuilder.build(), mCaptureCallback,mBackgroundHandler);} catch (CameraAccessException e) {e.printStackTrace();}

Android Camera HAL 3 architecture
and Camera2 API (Eddy Talvala and others)}/*** Capture a still picture. This method should be called when we get a response in* {@link #mCaptureCallback} from both {@link #lockFocus()}.
*/private void captureStillPicture() {try {final Activity activity = getActivity();if (null == activity || null == mCameraDevice) {return;}// This is the CaptureRequest.Builder that we use to take a picture.
final CaptureRequest.Builder captureBuilder =mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
captureBuilder.addTarget(mImageReader.getSurface());// Use the same AE and AF modes as the preview.
captureBuilder.set(CaptureRequest.CONTROL_AF_MODE,
CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
captureBuilder.set(CaptureRequest.CONTROL_AE_MODE,
CaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH);// Orientation
int rotation = activity.getWindowManager().getDefaultDisplay().getRotation();captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, ORIENTATIONS.get(rotation));CameraCaptureSession.CaptureCallback CaptureCallback= new CameraCaptureSession.CaptureCallback() {• allows control over the camera• not over the heterogeneous processors
on a mobile platform

14

open problem!

@Override
public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request,
TotalCaptureResult result) {Toast.makeText(getActivity(), "Saved: " + mFile, Toast.LENGTH_SHORT).show();
unlockFocus();}};mCaptureSession.stopRepeating();
mCaptureSession.capture(captureBuilder.build(), CaptureCallback, null);} catch (CameraAccessException e) {e.printStackTrace();}}/**© Marc Levoy

HDR in the Google camera app[to be published in Proc. SIGGRAPH Asia 2016]✦

capture a burst of under-exposed images• same exposure on all images in burst• avoids blowing out highlights✦align and merge• all images look similar, so alignment is more robust• SNR ∝ sqrt(N) where N = number of frames in burst• reduces noise in shadows✦

tonemap• boost shadows• squeeze 14-bit merged image into 8-bit for display

15• preserve local contrast at the expense of global contrast © Marc Levoy

HDR in the Google camera app[to be published in Proc. SIGGRAPH Asia 2016]✦

capture a burst of under-exposed images• same exposure on all images in burst• avoids blowing out highlights✦align and merge• all images look similar, so alignment is more robust• reduces noise in shadows✦

tonemap• boost shadows• enhance local contrast to avoid washed-out look16

© Marc Levoy

HDR in the Google camera app[to be published in Proc. SIGGRAPH Asia 2016]✦

capture a burst of under-exposed images
Mach bandillusion:• same exposure on allthe
imagesin burst
each wedge should appear• avoids blowing out highlights
brighter on its right side✦align and merge• all images look similar, so alignment is more robust• SNR ∝ sqrt(N) where N = number of frames in burst• reduces noise in shadows✦

tonemap• boost shadows• squeeze 14-bit merged image into 8-bit for display

17• preserve local contrast at the expense of global contrast © Marc Levoy

HDR in the Google camera app[to be published in Proc. SIGGRAPH Asia 2016]✦

capture a burst of under-exposed images
Mach bandillusion:• same exposure on allthe
imagesin burst
each wedge should appear• avoids blowing out highlights
brighter on its right side✦align and merge• all images look similar, so alignment is more robust• SNR ∝ sqrt(N) where N = number of frames in burst• reduces noise in shadows✦

tonemap• boost shadows• squeeze 14-bit merged image into 8-bit for display

18• preserve local contrast at the expense of global contrast © Marc Levoy

La Grande Jatte, Georges Seurat, 1884

HDR in the Google camera app[to be published in Proc. SIGGRAPH Asia 2016]✦

capture a burst of under-exposed images• same exposure on all images in burst• avoids blowing out highlights✦align and merge• all images look similar, so alignment is more robust
X• SNR ≈ sN), N = number of frames in burst✓• reduces noise in shadows✦

tonemap• boost shadows• squeeze 14-bit merged image into 8-bit for display

20• preserve local contrast at the expense of global contrast © Marc Levoy

single frame

HDR+

23

single frame

24

HDR+

single frame

HDR+

Nexus 6

iPhone 6S+

Nexus 6P

23 mins after sunset

Nexus 6

iPhone 6S+

Nexus 6P

37 mins after sunsetsingle frame

HDR+

WHAT IF THE SCENEWERE DARKER?

(apologies to xkcd)

Illuminance on a surface✦

power accumulating on a surface per unit area,
considering light arriving from all directions
P
E =A⎛ lumens ⎞
⎜⎝
⎟
m2 ⎠

(Reinhard)

32© Marc Levoy

Illuminance on a surface✦

power accumulating on a surface per unit area,
considering light arriving from all directions
P
E =
A

✦

⎛ lumens ⎞
⎜⎝
⎟
m2 ⎠

related units• 1 lux = 1 lumen / m2
• British unit is footcandle- 1 candela held 1 foot from surface- 1 footcandle = 10.764 lux

33© Marc Levoy

Lux levels
300 lux = office lighting at Google
150 lux = desk lighting at home
50 lux = average restaurant
20 lux = romantic restaurant
10 lux = finding socks that match

34© Marc Levoy

Lux levels
300 lux = office lighting at Google
150 lux = desk lighting at home
50 lux = average restaurant
20 lux = romantic restaurant
10 lux = finding socks that match
3 lux = outdoor street lighting
1 lux = limit of reading
1/2 lux = full moon
1/5 lux = can’t find my keys
35

1/10 lux = wouldn’t take a step without a flashlight© Marc Levoy

1/4 lux, iPhone 6S+

HDR+

SeeInTheDark, ~50 frames, handheld, real-timeOperation of SeeInTheDark✦capture 10-bit Bayer RAW frame• Nexus 6P capable of 12Mpix at 30fps, but I use only 8fps✦

minimal pre-processing• correct for fixed pattern noise and hot pixels• downsize 4× in X and Y (instead of demosaicing) to 1Mpix✦

track feature(s) from previous frame• 2D convolutions, then fit homography, affine, similarity, or

simple translation, with help from accelerometer and gyro✦

combine in GPU using 4 × 16-bit FBOs (for R,G,B,G)• warp new frame to accumulator and blend in using IIR filter• apply black level correction, lens shading, white balancing,color correction, tone mapping, gamma curve, sharpening

39© Marc Levoy

Fixed pattern noise

40✦

differences in readout among columns✦

rises with ISO but independent of exposure time

© Marc Levoy

Fixed pattern noise

41✦

differences in readout among columns✦

rises with ISO but independent of exposure time✦measure using short exposure, but one shot is too noisy© Marc Levoy

Fixed pattern noise

42✦

differences in readout among columns✦

rises with ISO but independent of exposure time✦measure using short exposure, but one shot is too noisy✦

accumulate many short exposures to build a corrector© Marc Levoy

Hot pixels

45✦

isolated pixels with excessive current leakage✦

rises with ISO and exposure time

© Marc Levoy

Removing hot pixels✦

46

just look for isolated bright pixels, right? No, too noisy!

© Marc Levoy

Removing hot pixels

47✦

just look for isolated bright pixels, right? No, too noisy!

✦

accumulate to reduce noise, then look for them

© Marc Levoy

Removing hot pixels✦

just look for isolated bright pixels, right? No, too noisy!

✦

accumulate to reduce noise, then look for them
Q. Need long exposure, but cell phones have no shutters.
If you see the scene, you’ll be fooled by sharp features.
What to do?
A. Accumulate during auto-focus sweep!

48© Marc Levoy

Removing hot pixels✦

just look for isolated bright pixels, right? No, too noisy!

✦

accumulate to reduce noise, then look
Q. Need long exposure, but cell phones have no shutters.
If you see the scene, you’ll be fooled by sharp features.
What to do?
A. Accumulate during auto-focus sweep!

✦

50

replace hot pixels with average of its neighbors© Marc Levoybefore hot pixel correction

after hot pixel correction

Operation of SeeInTheDark✦capture 10-bit Bayer RAW frame• Nexus 6P capable of 12Mpix at 30fps, but I use only 8fps✦

minimal pre-processing• correct for fixed pattern noise and hot pixels• downsize 4× in X and Y (instead of demosaicing) to 1Mpix✦

track feature(s) from previous frame• 2D convolutions, then fit homography, affine, similarity, or

simple translation, with help from accelerometer and gyro✦

combine in GPU using 4 × 16-bit FBOs (for R,G,B,G)• warp new frame to accumulator and blend in using IIR filter• apply black level correction, lens shading, white balancing,color correction, tone mapping, gamma curve, sharpening

53© Marc Levoyaperture problem

vignetting causes
loss of signal

text is a repeating texture

How much can the gyro help?
✦

MEMS gyros are accurate, but they drift• ~1 pixel / frame (for 1Mpix image@8fps)

✦

✦

64

yaw is worst, because it can’t be correctedusing the gravity vector from the accelerometerthe compass is too inaccurate to correct anything© Marc Levoy

after 1 second

after 15 seconds

Operation of SeeInTheDark✦capture 10-bit Bayer RAW frame• Nexus 6P capable of 12Mpix at 30fps, but I use only 8fps✦

minimal pre-processing• correct for fixed pattern noise and hot pixels• downsize 4× in X and Y (instead of demosaicing) to 1Mpix✦

track feature(s) from previous frame• 2D convolutions, then fit homography, affine, similarity, or

simple translation, with help from accelerometer and gyro✦

combine in GPU using 4 × 16-bit FBOs (for R,G,B,G)• warp new frame to accumulator and blend in using IIR filter• apply black level correction, lens shading, white balancing,color correction, tone mapping, gamma curve, sharpening67© Marc Levoyaccumulator after first framesecond framesecond frame warped to match accumulator

accumulator after blending in second frame

1 luxafter blending in many more framesTwo ways to average images✦

FIR filter

O(N) memory

y[n] = b0 x[n] + b1 x[n-1] + ... + bN x[n-N]• Example:
accum[n] = ⅓ frame[n] + ⅓ frame[n-1] + ⅓ frame[n-2]✦IIR filtery[n] = k1 x[n] + k2 y[n-1]O(1) memory• Example:
accum[n] = ½ frame[n] + ½ accum[n-1]= ½ frame[n] + ¼ frame[n-1] + ⅛ frame[n-2] + ...

73© Marc Levoy

single frame

1/2 frame[n] + 1/2 accum[n-1]1/4 frame[n] + 3/4 accum[n-1]1/8 frame[n] + 7/8 accum[n-1]1/16 frame[n] + 15/16 accum[n-1]1/32 frame[n] + 31/32 accum[n-1]1/64 frame[n] + 63/64 accum[n-1]1/2 lux

1/64 frame[n] + 63/64 accum[n-1]

HDR+

HDR+

SeeInTheDarkordinary cell phone pictureordinary cell phone picturesingle framefull-resolution single frame

WHAT IF THE SCENEWERE DARKER?

AE/AF sequence (30fps)

AE/AF sequence (6fps)

AE/AF sequence (6fps)

AE/AF sequence (6fps)

0.1 lux
accumulating for 5 seconds

full moon, single framefull moon, ~30 frames

gibbous moon 30º above horizon

gibbous moon 20º above horizon

WHAT IF THE SCENEWERE DARKER?

Signal-to-noise ratio
mean pixel value
µ
SNR =
=
standard deviation of pixel valueσ=✦

P Qe tP Qe t + D t + N

2
r

where
P = incident photon flux (photons/pixel/sec)
Qe = quantum efficiencyt = exposure time (sec)
D = dark current (electrons/pixel/sec), including hot pixels
Nr = read noise (rms electrons/pixel), including fixed pattern noise102(http://learn.hamamatsu.com/articles/ccdsnr.html)© Marc Levoy

1 shot at ISO 5400 (gain=10 DN/e)same, but 10x darker scene

10 shots

10 shots• signal = 100, σ = 32, σ2 = 1,000• read noise σ = 27, σ2 = 730• total σ = 42
• SNR = 2.5 : 1• signal = 1000, σ = 100, σ2 = 10,000• read noise σ = 85, σ2 = 7,300• total σ = 132• SNR = 7.5 : 1• signal = 10, σ = 10, σ2 = 100• read noise σ = 27, σ2 = 730• total σ = 29
• SNR = 1:3• signal = 100, σ = 10, σ2 = 1,000• read noise σ = 85, σ2 = 7,300• total σ = 91
• SNR = 1:1

Where does the game stop?
✦

longer accumulation required to overcome read noise✦

noise causes jitter in alignment, leading to blur in image✦

not obvious how to auto-focus in very noisy images✦

variation in noise sources over time✦scene doesn’t hold still that long✦ways to cheat• IR illumination• more cameras...

104© Marc Levoy

Arrays of cameras
Stanford

Light L16

fake
105

© Marc Levoy

Light field camera arrays
Google Jump
Jaunt
Lytro

106

© Marc Levoy

(Eddy Talvala, Nexus 6P, 240fps, 2015, available on YouTube at https://www.youtube.com/watch?v=UaG8DuvpOLU)

WHAT IF THE SCENEWERE DARKER?

Sensor sensor;
Shot low, med, high;low.exposure = 1/80.;
med.exposure = 1/20.;
high.exposure = 1/5.;
sensor.capture(low);sensor.capture(med);
sensor.capture(high);
Frame frames[3];frames[0] = sensor.getFrame();frames[1] = sensor.getFrame();frames[2] = sensor.getFrame();fused = mergeHDR(frames);