Patent ID: 12192607

DETAILED DESCRIPTION

Systems and methods for cameras with a reconfigurable lens are described below. As built, a single imaging head camera's functionality is limited by its sensor resolution and lens field of view. While many uses can be derived from such a singular configuration, a lot of other use cases are out of bounds. This in effect limits the usability of such a camera.

Reconfigurable lens cameras are described herein, in which, by adding an accessory lens in front of the main outer lens, or mother lens, optical parameters of a lens stack including the mother lens are significantly altered. For example, the resulting reconfigured lens may have a significantly different field of view and/or distortion than those of the mother lens. For example, the mother lens can work without an accessory lens to offer a specific set of modes and use cases, but also in combination with an accessory lens, to offer a different set of modes supporting different use cases.

When the mother lens is reconfigured with the accessory lens, the image capture device (e.g., a main camera including the mother lens) identifies the reconfigured lens using one or more of a variety of available sensing modalities. For example, proximity sensing (e.g., using a magnetically actuated switch) may be used to detect the presence of an accessory lens. For example, changes in imaging sensor coverage caused by the addition of the accessory lens may be detected to detect the presence of an accessory lens. The image capture device may then reconfigure an internal image processing pipeline to work with the new reconfigured lens to offer the user access to a new sets of modes. For example, these new modes may include new image processing configurations and tuning.

In addition to detection of the accessory lens as present covering the mother lens, the image capture device may also identify which accessory lens is mounted over the mother lens (e.g., where multiple accessory lenses are built to work with the mother lens and provide for additional use cases). In some implementations, it can be useful that specific accessory lens information is loaded in non-volatile storage present with the accessory lens as part of an accessory lens structure used to mount the accessory lens over the mother lens. Such information may include, for example, an accessory lens identifier, identification of a mother lens the accessory lens is designed to work with, accessory lens characteristics (e.g., optical parameters) and/or accessory lens calibration data. This information stored in the non-volatile memory may be read (e.g., using a communication interface built into the accessory lens structure) by the image capture device (e.g., a camera) and aid in the image capture device reconfiguring itself to work with the reconfigured lens.

FIGS.1A-Dare isometric views of an example of an image capture device100. The image capture device100may include a body102having a lens104structured on a front surface of the body102, various indicators on the front of the surface of the body102(such as LEDs, displays, and the like), various input mechanisms (such as buttons, switches, and touch-screen mechanisms), and electronics (e.g., imaging electronics, power electronics, etc.) internal to the body102for capturing images via the lens104and/or performing other functions. The image capture device100may be configured to capture images and video and to store captured images and video for subsequent display or playback.

The image capture device100may include various indicators, including LED lights106and LCD display108. The image capture device100may also include buttons110configured to allow a user of the image capture device100to interact with the image capture device100, to turn the image capture device100on, to operate latches or hinges associated with doors of the image capture device100, and/or to otherwise configure the operating mode of the image capture device100. The image capture device100may also include a microphone112configured to receive and record audio signals in conjunction with recording video.

The image capture device100may include an I/O interface114(e.g., hidden as indicated using dotted lines). As best shown inFIG.1B, the I/O interface114can be covered and sealed by a removable door115of the image capture device100. The removable door115can be secured, for example, using a latch mechanism115a(e.g., hidden as indicated using dotted lines) that is opened by engaging the associated button110as shown.

The removable door115can also be secured to the image capture device100using a hinge mechanism115b, allowing the removable door115to pivot between an open position allowing access to the I/O interface114and a closed position blocking access to the I/O interface114. The removable door115can also have a removed position (not shown) where the entire removable door115is separated from the image capture device100, that is, where both the latch mechanism115aand the hinge mechanism115ballow the removable door115to be removed from the image capture device100.

The image capture device100may also include another microphone116integrated into the body102or housing. The front surface of the image capture device100may include two drainage ports as part of a drainage channel118. The image capture device100may include an interactive display120that allows for interaction with the image capture device100while simultaneously displaying information on a surface of the image capture device100. As illustrated, the image capture device100may include the lens104that is configured to receive light incident upon the lens104and to direct received light onto an image sensor internal to the lens104.

The image capture device100ofFIGS.1A-Dincludes an exterior that encompasses and protects internal electronics. In the present example, the exterior includes six surfaces (i.e. a front face, a left face, a right face, a back face, a top face, and a bottom face) that form a rectangular cuboid. Furthermore, both the front and rear surfaces of the image capture device100are rectangular. In other embodiments, the exterior may have a different shape. The image capture device100may be made of a rigid material such as plastic, aluminum, steel, or fiberglass. The image capture device100may include features other than those described here. For example, the image capture device100may include additional buttons or different interface features, such as interchangeable lenses, cold shoes and hot shoes that can add functional features to the image capture device100, etc.

The image capture device100may include various types of image sensors, such as a charge-coupled device (CCD) sensors, active pixel sensors (APS), complementary metal-oxide semiconductor (CMOS) sensors, N-type metal-oxide-semiconductor (NMOS) sensors, and/or any other image sensor or combination of image sensors.

Although not illustrated, in various embodiments, the image capture device100may include other additional electrical components (e.g., an image processor, camera SoC (system-on-chip), etc.), which may be included on one or more circuit boards within the body102of the image capture device100.

The image capture device100may interface with or communicate with an external device, such as an external user interface device, via a wired or wireless computing communication link (e.g., the I/O interface114). The user interface device may, for example, be the personal computing device360described below with respect toFIG.3B. Any number of computing communication links may be used. The computing communication link may be a direct computing communication link or an indirect computing communication link, such as a link including another device or a network, such as the internet, may be used.

In some implementations, the computing communication link may be a Wi-Fi link, an infrared link, a Bluetooth (BT) link, a cellular link, a ZigBee link, a near field communications (NFC) link, such as an ISO/IEC 20643 protocol link, an Advanced Network Technology interoperability (ANT+) link, and/or any other wireless communications link or combination of links.

In some implementations, the computing communication link may be an HDMI link, a USB link, a digital video interface link, a display port interface link, such as a Video Electronics Standards Association (VESA) digital display interface link, an Ethernet link, a Thunderbolt link, and/or other wired computing communication link.

The image capture device100may transmit images, such as panoramic images, or portions thereof, to the user interface device (not shown) via the computing communication link, and the user interface device may store, process, display, or a combination thereof the panoramic images.

The user interface device may be a computing device, such as a smartphone, a tablet computer, a phablet, a smart watch, a portable computer, and/or another device or combination of devices configured to receive user input, communicate information with the image capture device100via the computing communication link, or receive user input and communicate information with the image capture device100via the computing communication link.

The user interface device may display, or otherwise present, content, such as images or video, acquired by the image capture device100. For example, a display of the user interface device may be a viewport into the three-dimensional space represented by the panoramic images or video captured or created by the image capture device100.

The user interface device may communicate information, such as metadata, to the image capture device100. For example, the user interface device may send orientation information of the user interface device with respect to a defined coordinate system to the image capture device100, such that the image capture device100may determine an orientation of the user interface device relative to the image capture device100.

Based on the determined orientation, the image capture device100may identify a portion of the panoramic images or video captured by the image capture device100for the image capture device100to send to the user interface device for presentation as the viewport. In some implementations, based on the determined orientation, the image capture device100may determine the location of the user interface device and/or the dimensions for viewing of a portion of the panoramic images or video.

The user interface device may implement or execute one or more applications to manage or control the image capture device100. For example, the user interface device may include an application for controlling camera configuration, video acquisition, video display, or any other configurable or controllable aspect of the image capture device100.

The user interface device, such as via an application, may generate and share, such as via a cloud-based or social media service, one or more images, or short video clips, such as in response to user input. In some implementations, the user interface device, such as via an application, may remotely control the image capture device100such as in response to user input.

The user interface device, such as via an application, may display unprocessed or minimally processed images or video captured by the image capture device100contemporaneously with capturing the images or video by the image capture device100, such as for shot framing, which may be referred to herein as a live preview, and which may be performed in response to user input. In some implementations, the user interface device, such as via an application, may mark one or more key moments contemporaneously with capturing the images or video by the image capture device100, such as with a tag, such as in response to user input.

The user interface device, such as via an application, may display, or otherwise present, marks or tags associated with images or video, such as in response to user input. For example, marks may be presented in a camera roll application for location review and/or playback of video highlights.

The user interface device, such as via an application, may wirelessly control camera software, hardware, or both. For example, the user interface device may include a web-based graphical interface accessible by a user for selecting a live or previously recorded video stream from the image capture device100for display on the user interface device.

The user interface device may receive information indicating a user setting, such as an image resolution setting (e.g., 3840 pixels by 2160 pixels), a frame rate setting (e.g., 60 frames per second (fps)), a location setting, and/or a context setting, which may indicate an activity, such as mountain biking, in response to user input, and may communicate the settings, or related information, to the image capture device100.

FIGS.2A-Billustrate another example of an image capture device200. The image capture device200includes a body202and two camera lenses204,206disposed on opposing surfaces of the body202, for example, in a back-to-back or Janus configuration.

The image capture device may include electronics (e.g., imaging electronics, power electronics, etc.) internal to the body202for capturing images via the lenses204,206and/or performing other functions. The image capture device may include various indicators such as an LED light212and an LCD display214.

The image capture device200may include various input mechanisms such as buttons, switches, and touchscreen mechanisms. For example, the image capture device200may include buttons216configured to allow a user of the image capture device200to interact with the image capture device200, to turn the image capture device200on, and to otherwise configure the operating mode of the image capture device200. In an implementation, the image capture device200includes a shutter button and a mode button. It should be appreciated, however, that, in alternate embodiments, the image capture device200may include additional buttons to support and/or control additional functionality.

The image capture device200may also include one or more microphones218configured to receive and record audio signals (e.g., voice or other audio commands) in conjunction with recording video.

The image capture device200may include an I/O interface220and an interactive display222that allows for interaction with the image capture device200while simultaneously displaying information on a surface of the image capture device200.

The image capture device200may be made of a rigid material such as plastic, aluminum, steel, or fiberglass. In some embodiments, the image capture device200described herein includes features other than those described. For example, instead of the I/O interface220and the interactive display222, the image capture device200may include additional interfaces or different interface features. For example, the image capture device200may include additional buttons or different interface features, such as interchangeable lenses, cold shoes and hot shoes that can add functional features to the image capture device200, etc.

FIG.2Cis a cross-sectional view of the image capture device200ofFIGS.2A-B. The image capture device200is configured to capture spherical images, and accordingly, includes a first image capture device224and a second image capture device226. The first image capture device224defines a first field-of-view228as shown inFIG.2Cand includes the lens204that receives and directs light onto a first image sensor230.

Similarly, the second image capture device226defines a second field-of-view232as shown inFIG.2Cand includes the lens206that receives and directs light onto a second image sensor234. To facilitate the capture of spherical images, the image capture devices224,226(and related components) may be arranged in a back-to-back (Janus) configuration such that the lenses204,206face in generally opposite directions.

The fields-of-view228,232of the lenses204,206are shown above and below boundaries236,238, respectively. Behind the first lens204, the first image sensor230may capture a first hyper-hemispherical image plane from light entering the first lens204, and behind the second lens206, the second image sensor234may capture a second hyper-hemispherical image plane from light entering the second lens206.

One or more areas, such as blind spots240,242may be outside of the fields-of-view228,232of the lenses204,206so as to define a “dead zone.” In the dead zone, light may be obscured from the lenses204,206and the corresponding image sensors230,234, and content in the blind spots240,242may be omitted from capture. In some implementations, the image capture devices224,226may be configured to minimize the blind spots240,242.

The fields-of-view228,232may overlap. Stitch points244,246, proximal to the image capture device200, at which the fields-of-view228,232overlap may be referred to herein as overlap points or stitch points. Content captured by the respective lenses204,206, distal to the stitch points244,246, may overlap.

Images contemporaneously captured by the respective image sensors230,234may be combined to form a combined image. Combining the respective images may include correlating the overlapping regions captured by the respective image sensors230,234, aligning the captured fields-of-view228,232, and stitching the images together to form a cohesive combined image.

A slight change in the alignment, such as position and/or tilt, of the lenses204,206, the image sensors230,234, or both, may change the relative positions of their respective fields-of-view228,232and the locations of the stitch points244,246. A change in alignment may affect the size of the blind spots240,242, which may include changing the size of the blind spots240,242unequally.

Incomplete or inaccurate information indicating the alignment of the image capture devices224,226, such as the locations of the stitch points244,246, may decrease the accuracy, efficiency, or both of generating a combined image. In some implementations, the image capture device200may maintain information indicating the location and orientation of the lenses204,206and the image sensors230,234such that the fields-of-view228,232, stitch points244,246, or both may be accurately determined, which may improve the accuracy, efficiency, or both of generating a combined image.

The lenses204,206may be laterally offset from each other, may be off-center from a central axis of the image capture device200, or may be laterally offset and off-center from the central axis. As compared to image capture devices with back-to-back lenses, such as lenses aligned along the same axis, image capture devices including laterally offset lenses may include substantially reduced thickness relative to the lengths of the lens barrels securing the lenses. For example, the overall thickness of the image capture device200may be close to the length of a single lens barrel as opposed to twice the length of a single lens barrel as in a back-to-back configuration. Reducing the lateral distance between the lenses204,206may improve the overlap in the fields-of-view228,232.

Images or frames captured by the image capture devices224,226may be combined, merged, or stitched together to produce a combined image, such as a spherical or panoramic image, which may be an equirectangular planar image. In some implementations, generating a combined image may include three-dimensional, or spatiotemporal, noise reduction (3DNR). In some implementations, pixels along the stitch boundary may be matched accurately to minimize boundary discontinuities.

FIGS.3A-Bare block diagrams of examples of image capture systems.

Referring first toFIG.3A, an image capture system300is shown. The image capture system300includes an image capture device310(e.g., a camera or a drone), which may, for example, be the image capture device200shown inFIGS.2A-C.

The image capture device310includes a processing apparatus312that is configured to receive a first image from a first image sensor314and receive a second image from a second image sensor316. The image capture device310includes a communications interface318for transferring images to other devices. The image capture device310includes a user interface320to allow a user to control image capture functions and/or view images. The image capture device310includes a battery322for powering the image capture device310. The components of the image capture device310may communicate with each other via the bus324.

The processing apparatus312may be configured to perform image signal processing (e.g., filtering, tone mapping, stitching, and/or encoding) to generate output images based on image data from the image sensors314and316. The processing apparatus312may include one or more processors having single or multiple processing cores. The processing apparatus312may include memory, such as a random-access memory device (RAM), flash memory, or another suitable type of storage device such as a non-transitory computer-readable memory. The memory of the processing apparatus312may include executable instructions and data that can be accessed by one or more processors of the processing apparatus312.

For example, the processing apparatus312may include one or more dynamic random access memory (DRAM) modules, such as double data rate synchronous dynamic random-access memory (DDR SDRAM). In some implementations, the processing apparatus312may include a digital signal processor (DSP). In some implementations, the processing apparatus312may include an application specific integrated circuit (ASIC). For example, the processing apparatus312may include a custom image signal processor.

The first image sensor314and the second image sensor316may be configured to detect light of a certain spectrum (e.g., the visible spectrum or the infrared spectrum) and convey information constituting an image as electrical signals (e.g., analog or digital signals). For example, the image sensors314and316may include CCDs or active pixel sensors in a CMOS. The image sensors314and316may detect light incident through a respective lens (e.g., a fisheye lens). In some implementations, the image sensors314and316include digital-to-analog converters. In some implementations, the image sensors314and316are held in a fixed orientation with respective fields of view that overlap.

The communications interface318may enable communications with a personal computing device (e.g., a smartphone, a tablet, a laptop computer, or a desktop computer). For example, the communications interface318may be used to receive commands controlling image capture and processing in the image capture device310. For example, the communications interface318may be used to transfer image data to a personal computing device. For example, the communications interface318may include a wired interface, such as a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, or a FireWire interface. For example, the communications interface318may include a wireless interface, such as a Bluetooth interface, a ZigBee interface, and/or a Wi-Fi interface.

The user interface320may include an LCD display for presenting images and/or messages to a user. For example, the user interface320may include a button or switch enabling a person to manually turn the image capture device310on and off. For example, the user interface320may include a shutter button for snapping pictures.

The battery322may power the image capture device310and/or its peripherals. For example, the battery322may be charged wirelessly or through a micro-USB interface.

The image capture system300may implement some or all of the techniques described in this disclosure, such as the process600described inFIG.6and/or the process700described inFIG.7.

Referring toFIG.3B, another image capture system330is shown. The image capture system330includes an image capture device340and a personal computing device360that communicate via a communications link350. The image capture device340may, for example, be the image capture device100shown inFIGS.1A-D. The personal computing device360may, for example, be the user interface device described with respect toFIGS.1A-D.

The image capture device340includes an image sensor342that is configured to capture images. The image capture device340includes a communications interface344configured to transfer images via the communication link350to the personal computing device360.

The personal computing device360includes a processing apparatus362that is configured to receive, using a communications interface366, images from the image sensor342. The processing apparatus362may be configured to perform image signal processing (e.g., filtering, tone mapping, stitching, and/or encoding) to generate output images based on image data from the image sensor342.

The image sensor342is configured to detect light of a certain spectrum (e.g., the visible spectrum or the infrared spectrum) and convey information constituting an image as electrical signals (e.g., analog or digital signals). For example, the image sensor342may include CCDs or active pixel sensors in a CMOS. The image sensor342may detect light incident through a respective lens (e.g., a fisheye lens). In some implementations, the image sensor342includes digital-to-analog converters. Image signals from the image sensor342may be passed to other components of the image capture device340via a bus346.

The communications link350may be a wired communications link or a wireless communications link. The communications interface344and the communications interface366may enable communications over the communications link350. For example, the communications interface344and the communications interface366may include an HDMI port or other interface, a USB port or other interface, a FireWire interface, a Bluetooth interface, a ZigBee interface, and/or a Wi-Fi interface. For example, the communications interface344and the communications interface366may be used to transfer image data from the image capture device340to the personal computing device360for image signal processing (e.g., filtering, tone mapping, stitching, and/or encoding) to generate output images based on image data from the image sensor342.

The processing apparatus362may include one or more processors having single or multiple processing cores. The processing apparatus362may include memory, such as RAM, flash memory, or another suitable type of storage device such as a non-transitory computer-readable memory. The memory of the processing apparatus362may include executable instructions and data that can be accessed by one or more processors of the processing apparatus362. For example, the processing apparatus362may include one or more DRAM modules, such as DDR SDRAM.

In some implementations, the processing apparatus362may include a DSP. In some implementations, the processing apparatus362may include an integrated circuit, for example, an ASIC. For example, the processing apparatus362may include a custom image signal processor. The processing apparatus362may exchange data (e.g., image data) with other components of the personal computing device360via a bus368.

The personal computing device360may include a user interface364. For example, the user interface364may include a touchscreen display for presenting images and/or messages to a user and receiving commands from a user. For example, the user interface364may include a button or switch enabling a person to manually turn the personal computing device360on and off. In some implementations, commands (e.g., start recording video, stop recording video, or capture photo) received via the user interface364may be passed on to the image capture device340via the communications link350.

The image capture device340and/or the personal computing device360may be used to implement some or all of the techniques described in this disclosure, such as the process600ofFIG.6and/or the process700described inFIG.7.

FIGS.4A and4Billustrate an example of a lens assembly400with (4B) and without (4A) an optional accessory lens470attached to alter the optical properties of the lens assembly400.

FIG.4Aillustrates a cross-sectional side view of an example of a lens assembly400including a mother lens420and an image sensor430. The lens assembly400includes a lens barrel410including multiple inner lenses411,412,413, and414; a mother lens420; and an image sensor430. For example, the lens assembly400may be implemented as part of an image capture device, such as the image capture device100ofFIGS.1A-1D, the image capture device200ofFIGS.2A-2B, the image capture device310ofFIG.3A, or the image capture device340ofFIG.3B.

The lens assembly400includes a lens barrel410in a body of an image capture device. The lens barrel410may be an integrated part of a body of an image capture device. The lens barrel410includes multiple inner lenses411,412,413, and414. In some implementations, at least one of the multiple inner lenses411,412,413, and414is curved. In the depicted example, the lens barrel410includes a curved inner lens412. The curved inner lens412may refract light propagating through the lens barrel410to focus the light for capture by the image sensor430. The lens barrel410includes a second curved inner lens414. For example, the inner lenses411,412,413, and414may be attached (e.g., using glue and/or ledges and flanges (not shown)) to inner walls of the lens barrel410. The inner lenses411,412,413, and414may be oriented to direct light from a first end of the lens barrel410, roughly parallel to an optical axis416of the lens barrel410to a second end of the lens barrel410, where the light may be detected by the image sensor430to capture an image.

The lens assembly400includes a mother lens420positioned at an opposite end of the lens barrel from the image sensor430. The mother lens420is an outer (L1) lens of the lens assembly400. The lens assembly400may be used to capture images with the mother lens420exposed as the outer lens. The lens assembly400, with the mother lens420as the outer lens, has optical parameters (e.g., a field of view and a distortion profile) that can be used to support a first set of image capture modes. An image capture device that includes the lens assembly400may include a processing apparatus (e.g., the processing apparatus312) that is configured to perform signal processing on images captured using the lens assembly400including applying a warp based on optical parameters of the lens assembly400. For example, the lens assembly400, with the mother lens420as the outer lens, may have a narrow field of view or a wide field of view, and it may have one of many possible distortion profiles (e.g., rectilinear, fisheye, or anamorphic).

The lens assembly400includes an image sensor430mounted within a body of an image capture device at a second end of the lens barrel410. The image sensor430may be configured to capture images based on light incident on the image sensor through the mother lens420and the inner lenses411,412,413, and414. The image sensor430may be configured to detect light of a certain spectrum (e.g., the visible spectrum or the infrared spectrum) and convey information constituting an image as electrical signals (e.g., analog or digital signals). For example, the image sensor430may include charge-coupled devices (CCDs) or active pixel sensors in complementary metal-oxide-semiconductor (CMOS). In some implementations, the image sensor430includes a digital-to-analog converter. For example, the image sensor430may be configured to capture image data using a plurality of selectable exposure times.

FIG.4Billustrates a cross-sectional side view of an example of a system450including an accessory lens structure470mounted over the lens assembly400including the mother lens420and the image sensor430. For example, the lens assembly400may be implemented as part of an image capture device, such as the image capture device100ofFIGS.1A-1D, the image capture device200ofFIGS.2A-2B, the image capture device310ofFIG.3A, or the image capture device340ofFIG.3B. The accessory lens structure470includes an accessory lens472and a retaining mechanism480configured to fasten the accessory lens472in a position covering the mother lens420in a first arrangement and configured to disconnect the accessory lens472from the image capture device in a second arrangement. The accessory lens structure470includes an interface ring474attached to a base of the accessory lens472and configured to support the accessory lens472in position over the mother lens420at the end of the lens barrel410when the retaining mechanism480is in the first arrangement. The system450includes an O-ring482for waterproofing. The system450includes a non-volatile memory490storing more than two bits of data that is integrated in the accessory lens structure470; and a communication interface492integrated in the accessory lens structure470. For example, the non-volatile memory490may store data that identifies the accessory lens472, which can be read, via the communications interface492, by a processing apparatus of the image capture device and used to configure an image processing pipeline of the image capture device to process images captured using the accessory lens472. For example, system450may implement some or all of the techniques described in this disclosure, such as the process600described inFIG.6and/or the process700described inFIG.7.

The accessory lens structure470includes an accessory lens472and a retaining mechanism480configured to fasten the accessory lens472in a position covering the mother lens420in a first arrangement and configured to disconnect the accessory lens472from the image capture device in a second arrangement. The accessory lens472augments optical properties of a lens stack over the image sensor that includes the mother lens420when the retaining mechanism is in the first arrangement. For example, the accessory lens472may be used to provide a different field of view (e.g., fisheye, narrow, portrait, macro, or telephoto) for the image capture device than the field of view with the mother lens420serving as the outer lens. For example, the accessory lens472may be used to provide a different optical distortion (e.g., rectilinear, fisheye, or anamorphic) for the image capture device than the optical distortion with the mother lens420serving as the outer lens. For example, the accessory lens472may augment the optical properties of the lens stack including the mother lens420in such a way as to facilitate robust image stabilization processing. For example, when the retaining mechanism480is in the first arrangement, a field of view of the lens stack may be projected as a circle within a detectable area of the image sensor430. In some implementations, the processing apparatus is configured to: access an image captured using the image sensor430when the retaining mechanism480is in the first arrangement; apply an electronic image stabilization rotation to the image to obtain a stabilized image; and crop the stabilized image to a rectangular output image from within the circle.

The retaining mechanism480may include a fastening mechanism configured to facilitate transition between the first arrangement and the second arrangement by removably fastening the retaining mechanism480to the lens barrel410or another nearby portion of the body of the image capture device. In the example depicted inFIG.4B, a threaded mechanism is employed to fasten the retaining mechanism480to the lens barrel410and fasten the accessory lens472in a position covering a first end of the lens barrel410. In some implementations (not shown inFIG.4B), a retaining mechanism480may employ other fastening mechanisms to secure a retaining ring to a body of an image capture device. For example, a retaining mechanism may include a bayonet mechanism (e.g., such as the bayonet mechanism described in relation toFIG.5A) configured to facilitate transition between the first arrangement and the second arrangement. For example, a retaining mechanism may include a threaded mechanism (e.g., such as the threaded mechanism described in relation toFIG.5B) configured to facilitate transition between the first arrangement and the second arrangement. For example, a retaining mechanism may include a snap-ring mechanism (e.g., such as the snap-ring mechanism described in relation toFIG.5C) configured to facilitate transition between the first arrangement and the second arrangement. For example, a retaining mechanism may include screw holes that enable screws to fasten the retaining ring to the body in the first arrangement (e.g., as described in relation toFIG.5D).

In some implementations, the retaining mechanism480is glued to the accessory lens472. In some implementations, the accessory lens472is secured in the retaining mechanism480as a captured mount, such that the accessory lens472may be rotated within the retaining mechanism480. For example, the accessory lens472and the retaining mechanism480may be interlocked (e.g., using a flange and slot interface around a circumference of the accessory lens472) and travel together but the accessory lens472may still be loose enough to turn inside the retaining mechanism480independently. In some implementations, the accessory lens472is firmly held in a fixed orientation in the first arrangement by a friction lock formed by pressing the retaining mechanism480against the accessory lens472in its position covering the first end of the lens barrel410.

The system450includes an O-ring482for waterproofing that is positioned radially around the accessory lens472. The O-ring may be composed of a rubbery material. For example, the O-ring482may be positioned to be compressed between the retaining mechanism480, the accessory lens472and the body (e.g., the lens barrel410) of the image capture device to form a waterproofing seal. In some implementations, the O-ring482may be glued to the retaining mechanism480and/or to the accessory lens472.

Although not shown inFIG.4B, the system450includes a processing apparatus (e.g., the processing apparatus312), integrated in the image capture device that includes the lens assembly400. The processing apparatus is configured to: receive data that is stored in the non-volatile memory490when the retaining mechanism is in the first arrangement. For example, the non-volatile memory490may store an identifier of the accessory lens472. For example, the non-volatile memory490may store calibration data of the accessory lens472. For example, the non-volatile memory490may store optical parameters of the accessory lens472. For example, the non-volatile memory490may include a read only memory (ROM). For example, the non-volatile memory490may include a flash drive.

In some implementations, the processing apparatus is configured to automatically detect when the retaining mechanism480is in the first arrangement. For example, sensing modalities, such as proximity sensing (e.g., using a capacitive coupling, using electrical contacts between the accessory lens structure470and the lens assembly400, and/or using a magnetically actuated switch in the image capture device and a magnet in the accessory lens structure470) and/or detecting changes in image sensor coverage caused by the accessory lens472, may be used automatically detect when the retaining mechanism480is in the first arrangement. In some implementations, the processing apparatus is configured to: determine a warp mapping based on the data received from the non-volatile memory490; and apply the warp mapping to an image captured using the image sensor430when the retaining mechanism480is in the first arrangement. In some implementations, the processing apparatus is configured to, responsive to detecting that the accessory lens472has been mounted, prompt a user, via a user interface (e.g., the user interface320), to confirm a lens configuration change. For example, a confirmation prompt may be presented in the interactive display120. In some implementations, the processing apparatus is configured to, responsive to detecting that the accessory lens472has been mounted, automatically identify the accessory lens472from among a set of multiple supported accessory lenses. For example, the accessory lens472may be identified based on data (e.g., a lens identifier) stored in the non-volatile memory490that is received by the processing apparatus.

The accessory lens structure470includes a communication interface492integrated in the accessory lens structure470. The communication interface492may be used to transfer data from the non-volatile memory490to the processing apparatus (e.g., the processing apparatus312). For example, the processing apparatus may be configured to receive data stored in the non-volatile memory490as signals transmitted via the communications interface492. In some implementations, the communications interface492includes a radio-frequency identification tag and the image capture device includes a radio-frequency identification reader configured to read signals from the radio-frequency identification tag. In some implementations, the communications interface492includes one or more electrical conductors configured to contact one or more corresponding electrical conductors on the image capture device when the retaining mechanism480is in the first arrangement. In some implementations, the communications interface492includes a capacitive coupling. In some implementations, the communications interface492includes an optical coupling.

In some implementations (not shown inFIGS.4A-4B), the mother lens420is also removeably attached to the lens barrel410. For example, this structure may allow the mother lens420to be replaced in the event that the mother lens420is scratched. For example, the mother lens420may be fastened at the end of the lens barrel using a fastening mechanism (e.g., a bayonet mechanism, a threaded mechanism, a snap-ring mechanism, or screws), and the accessory lens structure470may be fastened over the mother lens420using a fastening mechanism to attached the accessory lens structure470to the fastening mechanism of the mother lens420or another part of the body of the image capture device. In some implementations (not shown inFIGS.4A-4B), the accessory lens structure470includes multiple lens elements stacked over the mother lens420when the retaining mechanism480is in the first arrangement.

FIG.5Aillustrates an example of an accessory lens structure500including a bayonet mechanism. The accessory lens structure500includes a lens barrel510(e.g., similar to the lens barrel410ofFIG.4B), which may include a mother lens and one or more inner lenses that may be curved. The lens barrel510is part of a body of an image capture device that includes a male bayonet ring512, which may be attached to or otherwise integrated with the lens barrel510or another portion of the body. The accessory lens structure500includes a retaining ring514that includes a female bayonet ring. For example, the retaining ring514may be pushed onto the male bayonet ring of the body in an unlocked position and turned to a locked position to fasten the retaining ring514to the body and fasten an accessory lens516(e.g., the accessory lens472) in a position covering a first end of the lens barrel510. The accessory lens structure500includes an O-ring518that may be positioned radially around the accessory lens516and compressed between the retaining ring514and the accessory lens516and/or the body to waterproof the accessory lens structure500. The accessory lens structure500may offer advantages over alternative lens assemblies, such as robust reusability over many cycles of removing and replacing an accessory lens, over-center locking, an enhanced user experience (e.g., easy to remove/replace), and the retaining ring514maybe made sturdy by making the retaining ring514out of a strong metal. A drawback may be a relatively large outer diameter of the retaining ring514.

FIG.5Billustrates an example of an accessory lens structure520including a threaded mechanism. The accessory lens structure520includes a lens barrel530(e.g., similar to the lens barrel410ofFIG.4B), which may include a mother lens and one or more inner lenses that may be curved. The lens barrel530is part of a body of an image capture device that includes a male threaded interface534, which may be attached to or otherwise integrated with the lens barrel530or another portion of the body. The accessory lens structure520includes a retaining ring532that includes a female threaded interface. For example, the retaining ring532may be twisted onto the male threaded interface534of the body to fasten the retaining ring532to the body and fasten an accessory lens536(e.g., the accessory lens472) in a position covering a first end of the lens barrel530. The accessory lens structure520includes an O-ring538that may be positioned inside (e.g., vertically under) the accessory lens536and compressed between the accessory lens536and the body (e.g., the lens barrel530) to waterproof the accessory lens structure520. The accessory lens structure520may offer advantages over alternative lens assemblies, such as a low profile. A drawback may be a risk of cross threading or stripping.

FIG.5Cillustrates an example of an accessory lens structure540including a snap-ring mechanism. The accessory lens structure540includes a lens barrel550(e.g., similar to the lens barrel410ofFIG.4B), which may include a mother lens and one or more inner lenses that may be curved. The lens barrel550is part of a body of an image capture device that includes a male snap-ring interface554, which may be attached to or otherwise integrated with the lens barrel550or another portion of the body. The accessory lens structure540includes a retaining ring552that includes a female snap-ring interface. For example, the retaining ring552may be pushed onto the male snap-ring interface554of the body to fasten the retaining ring552to the body and fasten an accessory lens516(e.g., the accessory lens472) in a position covering a first end of the lens barrel550. The accessory lens structure540includes an O-ring558that may be positioned radially around the accessory lens556and compressed between the retaining ring552and the accessory lens556and/or the body to waterproof the accessory lens structure540. The accessory lens structure540may offer advantages over alternative lens assemblies, such as ease of installation. A drawback may be greater difficulty in removing the retaining ring552.

FIG.5Dillustrates an example of an accessory lens structure560including screw holes. The accessory lens structure560includes a lens barrel570(e.g., similar to the lens barrel410ofFIG.4B), which may include a mother lens and one or more inner lenses that may be curved. The lens barrel570is part of a body of an image capture device that includes screw holes574, which may be attached to or otherwise integrated with the lens barrel570or another portion of the body. The accessory lens structure560includes a retaining ring572that includes screw holes. For example, the retaining ring572may be fastened to body by driving screws580and582through the screw holes of the retaining ring572, into the screw holes574of the body to fasten an accessory lens516(e.g., the accessory lens472) in a position covering a first end of the lens barrel570. The accessory lens structure560includes an O-ring578that may be positioned radially around the accessory lens576and compressed between the retaining ring572and the accessory lens576and/or the body to waterproof the accessory lens structure560. The accessory lens structure560may offer advantages over alternative lens assemblies, such as robust fastening of the retaining ring572and the accessory lens576into position. A drawback may be a large size and poor aesthetics of the retaining ring572.

FIG.6is a flowchart of an example of a process600for using an accessory lens structure (e.g., the accessory lens structure470) with an image capture device. The process600includes automatically detecting that an accessory lens structure has been mounted to an image capture device including a mother lens and an image sensor configured to detect light incident through the mother lens, such that an accessory lens of the accessory lens structure is positioned covering the mother lens; responsive to detecting that the accessory lens structure has been mounted, automatically identifying the accessory lens from among a set of multiple supported accessory lenses; accessing an image captured using the image sensor when the accessory lens structure is positioned covering the mother lens; determining a warp mapping based on identification of the accessory lens; applying the warp mapping to the image to obtain a warped image; and transmitting, storing, or displaying an output image based on the warped image. For example, the process600may be implemented by an image capture device, such as the image capture device100ofFIGS.1A-ID, the image capture device200ofFIGS.2A-2B, the image capture device310ofFIG.3A, or the image capture device340ofFIG.3B.

The process600includes automatically detecting610that an accessory lens structure (e.g., the accessory lens structure470) has been mounted to an image capture device including a mother lens (e.g., the mother lens420) and an image sensor (e.g., the image sensor430) configured to detect light incident through the mother lens, such that an accessory lens (e.g., the accessory lens472) of the accessory lens structure is positioned covering the mother lens. Mounting of the accessory lens structure may be detected610using a variety of sensing modalities. In some implementations, automatically detecting610that the accessory lens structure has been mounted to the image capture device includes using a proximity sensor integrated in the image capture device. For example, a magnetically actuated switch may be integrated in the image capture device near the lens assembly including the mother lens, and the magnetically actuated switch may be used to detect610the presence of a magnet that is integrated in the accessory lens structure (e.g., in a plastic interface ring attached to a base of the accessory lens). For example, a capacitive coupling between a device integrated in the accessory lens structure and a device integrated in the image capture device may be used to detect610when the accessory lens structure has been mounted. For example, the accessory lens structure and a portion of the body of the image capture device (e.g., a lens barrel) may include respective electrical contacts that come into contact with each other when the accessory lens structure is mounted. In some implementations, automatically detecting610that the accessory lens structure has been mounted to the image capture device includes detecting a change in image sensor coverage of the image sensor. For example, the image sensor coverage may be monitored and when a change in the coverage corresponding to the mounting of the accessory lens structure may be detected610when the accessory lens structure is mounted. In some implementations, the process600includes, responsive to detecting610that the accessory lens has been mounted, prompting a user, via a user interface (e.g., the user interface320), to confirm a lens configuration change. Prompting a user to confirm the mounting may prevent false detections in some circumstances.

The process600includes, responsive to detecting that the accessory lens structure has been mounted, automatically identifying620the accessory lens from among a set of multiple supported accessory lenses. In some implementations, automatically identifying620the accessory lens includes receiving data from a non-volatile memory (e.g., the non-volatile memory490) integrated in the accessory lens structure. For example, the data from the non-volatile memory may include an identifier for the accessory lens or other identifying information, such as optical parameters or configuration parameters for an image signal processor. Data from the non-volatile memory may be received via communications interface integrated in the accessory lens structure with the non-volatile memory. For example, the accessory lens structure may include a radio frequency identification (RFID) tag, and the image capture device may include an RFID reader configured to read data from the non-volatile memory of the RFID tag. Other types of communication interfaces may be used to receive data from the non-volatile memory, such as a capacitive coupling, an optical coupling, one or more electrical conductors that meet at one or more electrical contacts between the accessory lens structure and the image capture device when the accessory lens structure is mounted. In some implementations, a bar code on the accessory lens structure is read by an optical bar code reader integrated in the image capture device when the accessory lens structure is mounted. In some implementations, automatically identifying620the accessory lens includes comparing image sensor coverage of the image sensor to an image sensor coverage profile associated with the accessory lens.

The process600includes accessing630an image (e.g., a still image or a frame of video) captured using the image sensor when the accessory lens structure is positioned covering the mother lens. For example, the input image may be accessed630from an image sensor (e.g., the image sensor430or the image sensor314) via a bus (e.g., the bus324). In some implementations, the input image may be accessed630via a communications link (e.g., the communications link350). For example, the input image may be accessed630via a wireless or wired communications interface (e.g., Wi-Fi, Bluetooth, USB, HDMI, Wireless USB, Near Field Communication (NFC), Ethernet, a radio frequency transceiver, and/or other interfaces). For example, the input image may be accessed630via communications interface366. For example, the input image may be accessed630via a front ISP that performs some initial processing on the accessed630input image. For example, the input image may represent each pixel value in a defined format, such as in a RAW image signal format, a YUV image signal format, or a compressed format (e.g., an MPEG or JPEG compressed bitstream). For example, the input image may be stored in a format using the Bayer color mosaic pattern.

The process600includes determining640a warp mapping based on identification of the accessory lens. For example, the warp mapping may include a lens distortion correction that is determined based on the identification of the accessory lens. For example, identification of the accessory lens may be made based on data read from a non-volatile memory integrated in the accessory lens structure. In some implementations, the data from the non-volatile memory includes parameters of the warp mapping (e.g., parameters of a lens distortion correction transformation) that are used to determine640the warp mapping. In some implementations, the data from the non-volatile memory includes optical parameters (e.g., optical parameters of the accessory lens or optical parameters of a lens stack including the accessory lens and the mother lens), and the warp mapping is determined640based on these optical parameters. For example, the warp mapping may include a series of transformations, such as lens distortion correction, electronic rolling shutter correction, and parallax correction (for implementations with using two or more image sensors).

The process600includes applying650the warp mapping to the image to obtain a warped image. For example, the warp mapping may specify the determination of image portions (e.g., pixels or blocks of pixels) of the warped image based on linear combinations of one or more corresponding image portions of the image and/or one or more corresponding image portions of an image captured with another image sensor of the image capture device.

For example, the accessory lens may be used to facilitate robust electronic image stabilization in some use cases. In some implementations, when the accessory lens structure is positioned covering the mother lens, a field of view of a lens stack including the accessory lens and the mother lens is projected as a circle within a detectable area of the image sensor. Although not explicitly shown inFIG.6, the process600may also include applying an electronic image stabilization rotation to the image to obtain a stabilized image; and cropping the stabilized image to a rectangular output image from within the circle (e.g., as described in relation to the process700ofFIG.7).

The process600includes transmitting, storing, or displaying660an output image based on the warped image. For example, the output image may be transmitted660to an external device (e.g., a personal computing device) for display or storage. For example, the output image may be the same as the warped image. For example, the output image may be a composite image determined by stitching an image based on the output image to one or more images from other image sensors with overlapping fields of view. For example, the output image may be an electronically stabilized image based on the warped image. For example, the output image may be compressed using an encoder (e.g., an MPEG encoder). For example, the output image may be transmitted660via the communications interface318. For example, the output image may be displayed660in the user interface320or in the user interface364. For example, the output image may be stored660in memory of the processing apparatus312or in memory of the processing apparatus362.

FIG.7is a flowchart of an example of a process700for improving electronic image stabilization using an accessory lens structure with an image capture device. The image capture device includes a mother lens (e.g., the mother lens420) and an image sensor (e.g., the image sensor430) configured to detect light incident through the mother lens. The accessory lens structure (e.g., the accessory lens structure470) includes an accessory lens (e.g., the accessory lens472) and a retaining mechanism (e.g., the retaining mechanism480) configured to fasten the accessory lens in a position covering the mother lens in a first arrangement and configured to disconnect the accessory lens from the image capture device in a second arrangement. In this example, the accessory lens augments optical properties of a lens stack over the image sensor that includes the mother lens, such that a field of view of the lens stack is projected as a circle within a detectable area of the image sensor, when the retaining mechanism is in the first arrangement. The process700includes accessing710an image captured using the image sensor when the retaining mechanism is in the first arrangement; applying720an electronic image stabilization rotation to the image to obtain a stabilized image; and cropping730the stabilized image to a rectangular output image from within the circle. For example, the process600may be implemented by an image capture device, such as the image capture device100ofFIGS.1A-1D, the image capture device200ofFIGS.2A-2B, the image capture device310ofFIG.3A, or the image capture device340ofFIG.3B.

For example, the image capture device may include a processing apparatus integrated in the image capture device. In some implementations, the processing apparatus is configured to automatically detect610when the retaining mechanism is in the first arrangement. In some implementations, the processing apparatus is configured to, responsive to detecting that the accessory lens structure has been mounted, prompt a user, via a user interface (e.g., the interactive display120), to confirm a lens configuration change. In some implementations, the processing apparatus is configured to, responsive to detecting that the accessory lens structure has been mounted, automatically identify620the accessory lens from among a set of multiple supported accessory lenses.

The process700includes accessing710an image (e.g., a still image or a frame of video) captured using the image sensor when the retaining mechanism is in the first arrangement. For example, the input image may be accessed710from an image sensor (e.g., the image sensor430or the image sensor314) via a bus (e.g., the bus324). In some implementations, the input image may be accessed710via a communications link (e.g., the communications link350). For example, the input image may be accessed710via a wireless or wired communications interface (e.g., Wi-Fi, Bluetooth, USB, HDMI, Wireless USB, Near Field Communication (NFC), Ethernet, a radio frequency transceiver, and/or other interfaces). For example, the input image may be accessed710via communications interface366. For example, the input image may be accessed710via a front ISP that performs some initial processing on the accessed710input image. For example, the input image may represent each pixel value in a defined format, such as in a RAW image signal format, a YUV image signal format, or a compressed format (e.g., an MPEG or JPEG compressed bitstream). For example, the input image may be stored in a format using the Bayer color mosaic pattern.

The process700includes applying720an electronic image stabilization rotation to the image to obtain a stabilized image. For example, the electronic image stabilization rotation may be applied720to a portion (e.g., pixel or block of pixels) of the image. For example, a portion of the stabilized image may be shifted to a new address or position within the stabilized image based on the electronic image stabilization rotation. For example, the electronic image stabilization rotation may be applied720to all portions of the image within the field of view. In some implementations, the electronic image stabilization rotation may be determined based on angular rate measurements for the image capture device including the image sensor used to capture the image. For example, the electronic image stabilization rotation may be determined based on motion sensor (e.g., gyroscope and/or accelerometer) measurements from a time associated with the capture of the image. Because the field of view of the lens stack is projected as a circle within a detectable area of the image sensor, the image may be well suited to the application of significant electronic image stabilization rotation, resulting in little or no distortion due to edge effects.

The process700includes cropping730the stabilized image to a rectangular output image from within the circle. For example, the stabilized image may be cropped730to a rectangular output image to conform to an image or video encoding format or a display format.

While the disclosure has been described in connection with certain embodiments, it is to be understood that the disclosure is not to be limited to the disclosed embodiments but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures as is permitted under the law.