Mobile camera system

Some embodiments include methods and/or systems for using multiple cameras to provide optical zoom to a user. Some embodiments include a first camera unit of a multifunction device capturing a first image of a first visual field. A second camera unit of the multifunction device simultaneously captures a second image of a second visual field. In some embodiments, the first camera unit includes a first optical package with a first focal length. In some embodiments, the second camera unit includes a second optical package with a second focal length. In some embodiments, the first focal length is different from the second focal length, and the first visual field is a subset of the second visual field. In some embodiments, the first image and the second image are preserved to a storage medium as separate data structures.

BACKGROUND

1. Technical Field

This disclosure relates generally to camera module components and more specifically to the use of multiple cameras for zoom functions in mobile devices.

2. Description of the Related Art

The advent of small, mobile multipurpose devices such as smartphones and tablet or pad devices has resulted in a need for high-resolution, small form factor cameras, capable of generating high levels of image quality, for integration in the devices.

Increasingly, as users rely on these multifunction devices as their primary cameras for day-to-day use, users demand features, such as zoom photography, that they have become accustomed to using in dedicated-purpose camera bodies. The zoom function is useful for capturing the details of a scene or alternatively capturing the context in which those details exist. The ability to change focal length to achieve zoom effects is sufficiently compelling to users of dedicated purpose cameras that it compels them to carry bags with an array of removable lenses, each of which weighs more and takes up more space than many common examples of a multifunction device, such as a phone.

Providing the zoom feature in a camera unit of a multifunction device has traditionally required moving mechanical parts that increase complexity and cost of the device. Such moving parts also reduce reliability of the device and take up valuable space inside the device, which puts the desire for zoom functions in direct conflict with the desire for smaller camera units that take up less space in the multifunction device.

SUMMARY OF EMBODIMENTS

Some embodiments include methods and/or systems for using multiple cameras to provide optical zoom to a user. Some embodiments include a first camera unit of a multifunction device capturing a first image of a first visual field. A second camera unit of the multifunction device simultaneously captures a second image of a second visual field. In some embodiments, the first camera unit includes a first optical package with a first focal length. In some embodiments, the second camera unit includes a second optical package with a second focal length. In some embodiments, the first focal length is different from the second focal length, and the first visual field is a subset of the second visual field. In some embodiments, the first image and the second image are preserved to a storage medium as separate data structures.

DETAILED DESCRIPTION

Introduction to Multiple Cameras for Optical Zoom

Some embodiments include methods and/or systems for using multiple cameras to provide optical zoom to a user. Some embodiments include a first camera unit of a multifunction device capturing a first image of a first visual field. A second camera unit of the multifunction device simultaneously captures a second image of a second visual field. In some embodiments, the first camera unit includes a first optical package with a first focal length. In some embodiments, the second camera unit includes a second optical package with a second focal length. In some embodiments, the first focal length is different from the second focal length, and the first visual field is a subset of the second visual field. In some embodiments, the first image and the second image are preserved to a storage medium as separate data structures.

In some embodiments, the first image and second image are of different media types. For example, in some embodiments, the first image is a moving image data structure captured at a first frame rate. In some embodiments, the second image is a moving image data structure captured at a second frame rate. In some embodiments, the second frame rate is faster than the first frame rate. In some embodiments, the first image is a still image taken at time t(0), and the second image is a moving image data structure captured over a time interval including t(0).

Some embodiments assign metadata to the first image and the second image a time indexing feature for establishing that the first image and the second image correspond as having been simultaneously captured or captured at overlapping time intervals. Some embodiments display the first image in a screen interface with a control for switching to display of the second image, and, responsive to an actuation of the control, display the second image in place of the first image. Some embodiments generate a synthetic intermediate image at least in part from data of the first image and data of the second image. In some embodiments, the synthetic intermediate image has a third focal length different from each of the first focal length and the second focal length, and the synthetic intermediate image has a third visual field different from each of the first visual field and the second visual field. Some embodiments preserve storage of the first image and data of the second image after creation of the synthetic intermediate image. In some embodiments, one camera may have a straight configuration and the other folded, where the folded configuration allows for a longer focal length to support the narrower field of view with a low F number (i.e., a lower ratio of the focal length of a camera lens to the diameter of the aperture being used for a particular shot).

Some embodiments generate a synthetic result image at least in part from data of the first image and data of the second image. In some embodiments, the synthetic intermediate image has is generated by enhancing the first image using data from the second image. Some embodiments display the first image and the second image in a shared screen interface.

Some embodiments include a camera system of a multifunction device. In some embodiments, the camera system includes a first camera unit of a multifunction device for capturing a first image of a first visual field and a second camera unit of the multifunction device for simultaneously capturing a second image of a second visual field. In some embodiments, the first camera unit includes a first optical package configured for a first focal length. In some embodiments, the second camera unit includes a second optical package configured for a second focal length. In some embodiments, the first focal length is different from the second focal length.

In some embodiments, the camera system includes a processing unit configured to assign to the first image and the second image a time indexing feature for establishing that the first image and the second image were simultaneously captured. In some embodiments, the first camera unit includes a lens having a folded lens configuration with a longer focal length than a lens of the second camera unit, and the second visual field is centered on a second visual axis aligned with a first visual axis on which the first visual field is centered.

In some embodiments, the first camera unit includes a lens having a longer focal length than a lens of the second camera unit, and the second visual field is centered on a second visual axis aligned with a first visual axis on which the first visual field is centered. In some embodiments, the first camera unit includes a first moveable lens and a first image sensor attached a chassis of the camera unit, the second camera unit includes a lens and a second image sensor moveably attached a chassis of the camera unit.

In some embodiments, the first camera unit includes a first moveable lens and a first image sensor attached a chassis of the camera unit, and the second camera unit includes a lens and a second image sensor moveably attached a chassis of the camera unit. In some embodiments, the first camera unit and the second camera unit include a first image processing pipeline and a second image processing pipeline, respectively.

In some embodiments, the first camera unit includes a first fixed lens and a first image sensor moveably attached a chassis of the camera unit, and the second camera unit includes a second fixed lens and a second image sensor moveably attached a chassis of the camera unit. In some embodiments, the second camera unit includes a second fixed lens aligned to share use of the first image sensor moveably attached the chassis of the camera unit.

Some embodiments include a non-transitory computer-readable storage medium, storing program instructions, computer-executable to implement a first camera unit of a multifunction device capturing a first image of a first visual field, and a second camera unit of the multifunction device simultaneously capturing a second image of a second visual field. In some embodiments, the first camera unit includes a first optical package with a first focal length, the second camera unit includes a second optical package with a second focal length, the first focal length is different from the second focal length, and the first visual field is a subset of the second visual field.

In some embodiments, the program instructions are further computer-executable to implement assigning metadata to the first image and the second image a time indexing feature for establishing that the first image and the second image correspond as having been simultaneously captured. In some embodiments, the program instructions are further computer-executable to implement displaying the first image in a screen interface with a control for switching to display of the second image, and responsive to an actuation of the control, displaying the second image in place of the first image.

In some embodiments, the program instructions are further computer-executable to implement generating a synthetic intermediate image from data of the first image and data of the second image. In some embodiments, the synthetic intermediate image has a third focal length different from each of the first focal length and the second focal length, and the synthetic intermediate image has a third visual field different from each of the first visual field and the second visual field. In some embodiments, the program instructions are further computer-executable to implement preserving storage of the first image and data of the second image after creation of the synthetic intermediate image. In some embodiments, the synthetic intermediate image has is generated by enhancing the first image using data from the second image. In some embodiments, the program instructions are further computer-executable to implement displaying the first image and the second image in a shared screen interface.

In some embodiments, the first image is a moving image data structure captured at a first frame rate. In some embodiments, the second image is a moving image data structure captured at a second frame rate. In some embodiments, the second frame rate is faster than the first frame rate. In some embodiments, the first image is a still image taken at time t(0), and the second image is a moving image data structure captured over a time interval including t(0).

Multifunction Device Examples

It will also be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the intended scope. The first contact and the second contact are both contacts, but they are not the same contact.

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA and/or music player functions. Other portable electronic devices, such as laptops, cameras, cell phones, or tablet computers, may also be used. It should also be understood that, in some embodiments, the device is not a portable communications device, but is a desktop computer with a camera. In some embodiments, the device is a gaming computer with orientation sensors (e.g., orientation sensors in a gaming controller). In other embodiments, the device is not a portable communications device, but is a camera.

The various applications that may be executed on the device may use at least one common physical user-interface device, such as the touch-sensitive surface. One or more functions of the touch-sensitive surface as well as corresponding information displayed on the device may be adjusted and/or varied from one application to the next and/or within a respective application. In this way, a common physical architecture (such as the touch-sensitive surface) of the device may support the variety of applications with user interfaces that are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices with cameras.FIG.1is a block diagram illustrating portable multifunction device100with cameras164a-bin accordance with some embodiments. Cameras164a-bare sometimes called “optical sensors” for convenience, and may also be known as or called an optical sensor system. Device100may include memory102(which may include one or more computer readable storage mediums), memory controller122, one or more processing units (CPU's)120, peripherals interface118, RF circuitry108, audio circuitry110, speaker111, touch-sensitive display system112, microphone113, input/output (I/O) subsystem106, other input or control devices116, and external port124. Device100may include optical sensors164a-b. These components may communicate over one or more communication buses or signal lines103.

It should be appreciated that device100is only one example of a portable multifunction device, and that device100may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of the components. The various components shown in various of the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

Memory102may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory102by other components of device100, such as CPU120and the peripherals interface118, may be controlled by memory controller122.

In some embodiments, peripherals interface118, CPU120, and memory controller122may be implemented on a single chip, such as chip104. In some other embodiments, they may be implemented on separate chips.

I/O subsystem106couples input/output peripherals on device100, such as touch screen112and other input control devices116, to peripherals interface118. I/O subsystem106may include display controller156and one or more input controllers160for other input or control devices. The one or more input controllers160receive/send electrical signals from/to other input or control devices116. The other input control devices116may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate embodiments, input controller(s)160may be coupled to any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse. The one or more buttons (e.g.,208,FIG.2) may include an up/down button for volume control of speaker111and/or microphone113. The one or more buttons may include a push button (e.g.,206,FIG.2).

Touch-sensitive display112provides an input interface and an output interface between the device and a user. Display controller156receives and/or sends electrical signals from/to touch screen112. Touch screen112displays visual output to the user. The visual output may include graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some embodiments, some or all of the visual output may correspond to user-interface objects.

Device100may also include optical sensors or cameras164a-b. Optical sensors164a-bmay include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Optical sensors164a-breceive light from the environment, projected through one or more lens, and converts the light to data representing an image. In conjunction with imaging module143(also called a camera module), optical sensors164a-bmay capture still images or video. In some embodiments, an optical sensor is located on the back of device100, opposite touch screen display112on the front of the device, so that the touch screen display may be used as a viewfinder for still and/or video image acquisition. In some embodiments, another optical sensor is located on the front of the device so that the user's image may be obtained for videoconferencing while the user views the other video conference participants on the touch screen display. In embodiments in which multiple cameras or optical sensors164a-bare supported, each of the multiple cameras or optical sensors164a-bmay include its own photo sensor(s), or the multiple cameras or optical sensors164a-bmay be supported by a shared photo sensor. Likewise, in embodiments in which multiple cameras or optical sensors164a-bare supported, each of the multiple cameras or optical sensors164a-bmay include its own image processing pipeline of processors and storage units, or the multiple cameras or optical sensors164a-bmay be supported by a image processing pipeline of processors and storage units.

Device100may also include one or more proximity sensors166.FIG.28shows proximity sensor166coupled to peripherals interface118. Alternately, proximity sensor166may be coupled to input controller160in I/O subsystem106. In some embodiments, the proximity sensor turns off and disables touch screen112when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

Device100includes one or more orientation sensors168. In some embodiments, the one or more orientation sensors include one or more accelerometers (e.g., one or more linear accelerometers and/or one or more rotational accelerometers). In some embodiments, the one or more orientation sensors include one or more gyroscopes. In some embodiments, the one or more orientation sensors include one or more magnetometers. In some embodiments, the one or more orientation sensors include one or more of global positioning system (GPS), Global Navigation Satellite System (GLONASS), and/or other global navigation system receivers. The GPS, GLONASS, and/or other global navigation system receivers may be used for obtaining information concerning the location and orientation (e.g., portrait or landscape) of device100. In some embodiments, the one or more orientation sensors include any combination of orientation/rotation sensors.FIG.1shows the one or more orientation sensors168coupled to peripherals interface118. Alternately, the one or more orientation sensors168may be coupled to an input controller160in I/O subsystem106. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on an analysis of data received from the one or more orientation sensors.

Contact/motion module130may detect contact with touch screen112(in conjunction with display controller156) and other touch sensitive devices (e.g., a touchpad or physical click wheel). Contact/motion module130includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact/motion module130receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, may include determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations may be applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., “multitouch”/multiple finger contacts). In some embodiments, contact/motion module130and display controller156detect contact on a touchpad.

Contact/motion module130may detect a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns. Thus, a gesture may be detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (lift off) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (lift off) event.

Graphics module132includes various known software components for rendering and displaying graphics on touch screen112or other display, including components for changing the intensity of graphics that are displayed. As used herein, the term “graphics” includes any object that can be displayed to a user, including without limitation text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations and the like.

Text input module134, which may be a component of graphics module132, provides soft keyboards for entering text in various applications (e.g., contacts137, e-mail140, IM141, browser147, and any other application that needs text input).

Examples of other applications136that may be stored in memory102include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

In conjunction with RF circuitry108, audio circuitry110, speaker111, microphone113, touch screen112, display controller156, optical sensors164a-b, optical sensor controller158, contact module130, graphics module132, text input module134, contact list137, and telephone module138, videoconferencing module139includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.

In conjunction with touch screen112, display controller156, contact module130, graphics module132, text input module134, and dual camera module143, image management module144includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images.

In conjunction with RF circuitry108, touch screen112, display system controller156, contact module130, graphics module132, text input module134, e-mail client module140, and browser module147, calendar module148includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to do lists, etc.) in accordance with user instructions.

In conjunction with RF circuitry108, touch screen112, display system controller156, contact module130, graphics module132, text input module134, and browser module147, the widget creator module150may be used by a user to create widgets (e.g., turning a user-specified portion of a web page into a widget).

In conjunction with touch screen112, display controller156, contact module130, graphics module132, and text input module134, notes module153includes executable instructions to create and manage notes, to do lists, and the like in accordance with user instructions.

In conjunction with touch screen112, display system controller156, contact module130, graphics module132, audio circuitry110, speaker111, RF circuitry108, text input module134, e-mail client module140, and browser module147, online video module155includes instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen or on an external, connected display via external port124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module141, rather than e-mail client module140, is used to send a link to a particular online video.

FIG.2illustrates a portable multifunction device100having a touch screen112in accordance with some embodiments. The touch screen may display one or more graphics within user interface (UI)200. In this embodiment, as well as others described below, a user may select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers202(not drawn to scale in the figure) or one or more styluses203(not drawn to scale in the figure).

Device100may also include one or more physical buttons, such as “home” or menu button204. As described previously, menu button204may be used to navigate to any application136in a set of applications that may be executed on device100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touch screen112.

In one embodiment, device100includes touch screen112, menu button204, push button206for powering the device on/off and locking the device, volume adjustment button(s)208, Subscriber Identity Module (SIM) card slot210, head set jack212, and docking/charging external port124. Push button206may be used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process. In an alternative embodiment, device100also may accept verbal input for activation or deactivation of some functions through microphone113.

It should be noted that, although many of the examples herein are given with reference to optical sensors/cameras164a-b(on the front of a device), a rear-facing camera or optical sensor that is pointed opposite from the display may be used instead of or in addition to an optical sensors/cameras164a-bon the front of a device.

FIG.3Aillustrates a view of an example embodiment of camera module components arranged for multiple visual fields usable for a multiple camera system for portable zoom, according to at least some embodiments. A portable multifunction device3080includes a first optical sensor/camera unit3082with a first focal length3090for capturing a first visual field3088and a second optical sensor/camera unit3084with a first focal length3092for capturing a second visual field3086.

Some embodiments include a first camera unit3082(such as one of the cameras described below with respect toFIG.3BandFIG.3C) of a multifunction device capturing a first image of a first visual field3088. A second camera unit3084(such as one of the cameras described below with respect toFIG.3BandFIG.3C) of the multifunction device3080simultaneously captures a second image of a second visual field3086. In some embodiments, the first camera unit3082includes a first optical package with a first focal length3090. In some embodiments, the second camera unit3084includes a second optical package (described below with respect toFIG.3B, below) with a second focal length3092. In some embodiments, the first focal length3090is different from the second focal length3092, and the first visual field3088is a subset of the second visual field3086. In some embodiments, the first image and the second image are preserved to a storage medium as separate data structures.

Some embodiments assign metadata to the first image of the first visual field3088and the second image of the second visual field3086a time indexing feature for establishing that the first image of the first visual field3088and the second image of the second visual field3086correspond as having been simultaneously captured. Some embodiments display the first image of the first visual field3088in a screen interface with a control for switching to display of the second image of the second visual field3086, and, responsive to an actuation of the control, display the second image of the second visual field3086in place of the first image. Some embodiments generate a synthetic intermediate image at least in part from data of the first image of the first visual field3088and data of the second image of the second visual field3086. In some embodiments, the synthetic intermediate image has a third focal length different from each of the first focal length3090and the second focal length3092, and the synthetic intermediate image has a third visual field different from each of the first visual field3088and the second visual field3086. Some embodiments preserve storage of the first image of the first visual field3088and data of the second image of the second visual field3086after creation of the synthetic intermediate image.

Some embodiments generate a synthetic result image at least in part from data of the first image of the first visual field3088and data of the second image of the second visual field3086. In some embodiments, the synthetic intermediate image has is generated by enhancing the first image of the first visual field3088using data from the second image of the second visual field3086. Some embodiments display the first image of the first visual field3088and the second image of the second visual field3086in a shared screen interface.

Some embodiments include a camera system of a multifunction device. In some embodiments, the camera system includes a first camera unit3082of a multifunction device3080for capturing a first image of a first visual field3088and a second camera unit of the multifunction device for simultaneously capturing a second image of a second visual field3086. In some embodiments, the first camera unit3082includes a first optical package configured for a first focal length3090. In some embodiments, the second camera unit3084includes a second optical package configured for a second focal length3092. In some embodiments, the first focal length3090is different from the second focal length3092.

In some embodiments, the camera system includes a processing unit configured to assign to the first image of a first visual field3088and the second image a time indexing feature for establishing that the first image and the second image of a second visual field3086were simultaneously captured. In some embodiments, the first camera unit3082includes a lens having a folded lens configuration (not shown) with a longer focal length3090than a focal length3092of a lens of the second camera unit3084, and the second visual field3086is centered on a second visual axis aligned with a first visual axis on which the first visual field3088is centered. In some embodiments, the first camera unit3082includes a first moveable lens (shown below with respect toFIG.3B) and a first image sensor attached a chassis of the camera unit, the second camera unit includes a lens and a second image sensor moveably attached a chassis of the second camera unit3084.

In some embodiments, the first camera unit3082includes a first moveable lens and a first image sensor attached a chassis of the first3082camera unit, and the second camera unit3084includes a lens (shown below with respect toFIG.3B) and a second image sensor moveably attached a chassis of the second camera unit. In some embodiments, the first camera unit3082and the second camera unit3084include a first image processing pipeline and a second image processing pipeline, respectively.

In some embodiments, the first camera unit3082includes a first fixed lens and a first image sensor moveably attached a chassis of the first camera unit3082, and the second camera unit3084includes a second fixed lens and a second image sensor moveably attached a chassis of the second camera unit3084. In some embodiments, the second camera unit3084includes a second fixed lens aligned to share use of the first image sensor moveably attached the chassis of the second camera unit3084.

In some embodiments, the first image and second image are of different media types. For example, in some embodiments, the first image is a moving image data structure captured at a first frame rate. In some embodiments, the second image is a moving image data structure captured at a second frame rate. In some embodiments, the second frame rate is faster than the first frame rate. In some embodiments, the first image is a still image taken at time t(0), and the second image is a moving image data structure captured over a time interval including t(0).

In some embodiments, the first image has a first resolution and the second image has a second resolution. An example of the use of a first image that is a moving image data structure at a first frame rate and a second image that is a moving image data structure at a second frame rate arises in that some embodiments include second camera module3084recording 720p (also known as 720 pixels of vertical resolution progressive scan) slow motion video at 240 frames per second while first camera module3082is capturing 4K (horizontal resolution on the order of 4,000 pixels) video at 30 frames per second. In some embodiments, the analog-to-digital converter bandwidth required for each separate module to achieve the recording is 220-270 Mpixels/s. Achieving the same functionality with conventional single camera module technology requires up to 32 times higher analog-to-digital converter bandwidth for a single camera module if it is compared to embodiments in which there is a 2× difference in focal length from wide to tele module, providing benefits in terms of power, thermal dissipation, storage bandwidth, storage capacity, and actual achievable frame rates combined with zoom capability.

A use case for some embodiments is well-illustrated with respect to sports photography. In one example use case, it is possible to imagine a user of portable multifunction device3080filming a batter in a baseball game. Recording video of the game with portable multifunction device3080from bleachers, not shown, a user may decide to zoom in to capture a batter swinging and hitting the ball in slow motion using second camera module3084recording 720p slow motion video at 240 frames per second, but may subsequently want to switch to the simultaneously captured 4K video from first camera module3082at 30 frames per second of resulting home run in high quality video of the full baseball field, to capture the moments where the opposing team scrambles to catch the ball and the batter is running from base to base. Some embodiments enable this mixed-video capture by simultaneously recording using second camera module3084as a telephoto camera module in a 240 frames per second slow motion mode while at the same time using first camera module3082as a wide camera module in a 4K at 30 frames per second. After capturing a data structure including both video streams the awesome moment, some embodiments provide for a mixed-video data structure and an interface for the video streams from the two separate camera modules to be manually or automatically edited and combined to create a more engaging media which may contain normal 1080p video, 4K high resolution video, 720p motion video, and still images. In the example described above, this mixed-video media both captures the close up expressions of players, the peak action in slow motion, and frames it all in the context of a great play in a baseball game.

Another example of a use case for some embodiments arises in the context of capturing a child extinguishing candles on a birthday cake. In such an example, one can imagine a child about to blow out the candles on the birthday cake while all her friends are singing a birthday song. In some embodiments, second camera module3084can be used as a telephoto camera module to zoom in on the face of the child as she is about to blow out the candles and first camera module3082can capture a burst of high resolution still images of her smiling face. In some embodiments, first camera module3082is simultaneously capturing standard 1080p 30 frames per second video of the entire group of kids gathered and singing around the cake. Some embodiments provide an editing interface for combining the video stream from the wide camera module, either manually or automatically, with the close up portraits to create a much more engaging media experience which can be shared. As the two camera modules are synchronized in time, the still images can easily be automatically inserted at the right time in a final video stream.

FIG.3Bdepicts a side view of an example embodiment of camera module, according to at least some embodiments. Camera module3000, which is an embodiment of cameras164a-b, discussed below includes camera components such as an optics module (e.g., a lens barrel)3002attached to an optics holder3003and a magnet holder3006. An image sensor3070, which may or may not be mounted on a substrate that is not shown separately inFIG.3, is attached to a camera module base3008. The camera components may further include, in addition to components such as power and remote control connections not shown, a cover3012and suspension wires3020.

Optics module3002may be suspended on the base assembly3008by suspension of the upper springs3030and the suspension wires3020. Camera components may include one or more of, but are not limited to, optics3002, optics holder3003, magnet holder(s)3006, upper spring(s)3030, and lower spring(s)3032. The upper and lower spring(s) may be collectively referred to herein as optics springs. An optics module (e.g., a lens or lens assembly or lens barrel)3002may be screwed, mounted or otherwise held in or by an optics holder3003. In at least some embodiments, the optics3002/optics holder3003assembly may be suspended from or attached to the magnet holder3006by upper spring(s)3030, and lower spring(s)3032. Note that upper spring(s)3030and lower spring(s)3032are flexible to allow the optics assembly3000a range of motion along the Z (optical) axis for optical focusing, wires3020are flexible to allow a range of motion on the XY plane orthogonal to the optical axis for optical image stabilization.

Note that, in some embodiments, a camera may not include magnets and magnet holder(s)3006, but may include a yoke or other structure3006that may be used to help support the optics assembly on suspension wires3020via upper springs3030. In general, other embodiments of an optics assembly3000may include fewer or more components than the example optics assembly3000shown inFIG.3. Also note that, while embodiments show the optics assembly3000suspended on wires3020, other mechanisms may be used to suspend an optics assembly3000in other embodiments.

The autofocus yoke (e.g., magnet holder(s)3006) acts as the support chassis structure for the autofocus mechanism of actuator3000. The lens carrier (optics holder3003) is suspended on the autofocus yoke by an upper autofocus (AF) spring3030and a lower optics spring3032. In this way when an electric current is applied to the autofocus coil, Lorentz forces are developed due to the presence of the four magnets, and a force substantially parallel to the optical axis is generated to move the lens carrier, and hence lens, along the optical axis, relative to the support structure of the autofocus mechanism of the actuator, so as to focus the lens. In addition to suspending the lens carrier and substantially eliminating parasitic motions, the upper spring3030and lower spring3032also resist the Lorentz forces, and hence convert the forces to a displacement of the lens. This basic architecture shown inFIG.3is typical of some embodiments, in which optical image stabilization function includes moving the entire autofocus mechanism of the actuator (supported by the autofocus yoke) in linear directions orthogonal to the optical axis, in response to user handshake, as detected by some means, such a two or three axis gyroscope, which senses angular velocity. The handshake of interest is the changing angular tilt of the camera in ‘pitch and yaw directions’, which can be compensated by said linear movements of the lens relative to the image sensor.

In at least some embodiments, the suspension of the autofocus mechanism on the actuator3000support structure may be achieved by the use of four corner wires3020, for example wires with a circular cross-section. Each wire3020acts as a flexure beams capable of bending with relatively low stiffness, thus allowing motion in both optical image stabilization degrees-of-freedom. However, wire3020is in some embodiments relatively stiff in directions parallel to the optical axis, as this would require the wire to stretch or buckle, thus substantially preventing parasitic motions in these directions. In addition, the presence of four such wires, appropriately separated allows them to be stiff in the parasitic tilt directions of pitch and yaw, thus substantially preventing relative dynamic tilt between the lens and image sensor. This may be seen by appreciating that each wire3020is stiff in directions that require it to change in length, and hence the fixed points at the ends of each wire (eight points in total) will substantially form the vertices of a parallelepiped for all operational positions of the optical image stabilization mechanism.

FIG.3Cdepicts a folded-optics camera configuration for use with in portable multifunction device in accordance with some embodiments.FIG.3Cshows a cross-section through the optical stack of the camera (image capture device package307), an image sensor substrate assembly327, and a lens group337, a mirror357. Mirror357is used to fold the optics, and is, in some embodiments, nominally mounted at 45 degrees to the optical axis of all the lens elements. As can be observed inFIG.3C, the substrate assembly327includes a sensor377. A ceramic substrate387is also shown. Note that, in some embodiments, multiple substrates387and image sensors377are included.

FIG.4illustrates a user interface for a multiple camera system for portable zoom, according to at least some embodiments. A portable multifunction device400displays a first image of a first visual field404captured by a first camera unit and a second image of a second visual field402simultaneously captured by a second camera unit of the multifunction device400. A zoom control406is displayed within first image of a first visual field404. In the embodiment shown, zoom control406is an area of first image of first visual field404, that, in response to control actuation through the touch screen of portable multifunction device400, is used as a control for toggling the display mode for displaying first image of a first visual field404captured by a first camera unit and second image of a second visual field402simultaneously captured by a second camera unit of the multifunction device400.

Some embodiments assign metadata to the first image404and the second image402for a time indexing feature for establishing that the first image404and the second image402correspond as having been simultaneously captured. Some embodiments display the first image404in a screen interface with a control (e.g., similar to control406) for switching to display of the second image402, and, responsive to an actuation of the control406, display the second image402in place of the first image404. Some embodiments generate a synthetic intermediate image at least in part from data of the first image404and data of the second image402.

FIG.5is a flow chart of a method usable in a multiple camera system for portable zoom, according to at least some embodiments. A first camera unit of a multifunction device, having a first optical package with a first focal length, captures a first image of a first visual field (block500). A second camera unit of the multifunction device, having a second optical package with a second focal length different from the first focal length simultaneously captures a second image of a second visual field that is a subset of the first visual field (block502). The first image and the second image are preserved to a storage medium as separate data structures (block504).

FIG.6is a flow chart of a method usable in a multiple camera system for portable zoom, according to at least some embodiments. A first camera unit of a multifunction device, having a first optical package with a first focal length, captures a first image of a first visual field (block600). A second camera unit of the multifunction device, having a second optical package with a second focal length different from the first focal length simultaneously captures a second image of a second visual field that is a subset of the first visual field (block602). The first image and the second image are preserved to a storage medium as separate data structures (block604). Metadata is assigned to the first image and the second image a time indexing feature for establishing that the first image and the second image correspond as having been simultaneously captured (block606).

FIG.7is a flow chart of a method usable in a multiple camera system for portable zoom, according to at least some embodiments. A first camera unit of a multifunction device, having a first optical package with a first focal length, captures a first image of a first visual field (block700). A second camera unit of the multifunction device, having a second optical package with a second focal length different from the first focal length simultaneously captures a second image of a second visual field that is a subset of the first visual field (block702). The first image and the second image are preserved to a storage medium as separate data structures (block704). The first image is displayed in a screen interface with a control for switching to display of the second image (block706). Responsive to an actuation of the control, the second image is displayed in place of the first image (block708).

FIG.8is a flow chart of a method usable in a multiple camera system for portable zoom, according to at least some embodiments. A first camera unit of a multifunction device, having a first optical package with a first focal length, captures a first image of a first visual field (block800). A second camera unit of the multifunction device, having a second optical package with a second focal length different from the first focal length simultaneously captures a second image of a second visual field that is a subset of the first visual field (block802). The first image and the second image are preserved to a storage medium as separate data structures (block804). A synthetic intermediate image is generated, at least in part from data of the first image and data of the second image (block806). Storage of the first image and data of the second image is preserved after creation of the synthetic intermediate image (block808).

FIG.9is a flow chart of a method usable in a multiple camera system for portable zoom, according to at least some embodiments. A first camera unit of a multifunction device, having a first optical package with a first focal length, captures a first image of a first visual field (block900). A second camera unit of the multifunction device, having a second optical package with a second focal length different from the first focal length simultaneously captures a second image of a second visual field that is a subset of the first visual field (block902). The first image and the second image are preserved to a storage medium as separate data structures (block904). A synthetic intermediate image is generated, at least in part from data of the first image and data of the second image by enhancing the first image using data from the second image (block906).

FIG.10is a flow chart of a method usable in a multiple camera system for portable zoom, according to at least some embodiments. A first camera unit of a multifunction device, having a first optical package with a first focal length, captures a first image of a first visual field (block1000). A second camera unit of the multifunction device, having a second optical package with a second focal length different from the first focal length simultaneously captures a second image of a second visual field that is a subset of the first visual field (block1002). The first image and the second image are preserved to a storage medium as separate data structures (block1004). The first image and the second image are displayed in a shared screen interface (block1006).

Example Computer System

Various embodiments of a camera system as described herein, including embodiments of single frame camera active optical tilt alignment correction, as described herein may be executed in one or more computer systems1100, which may interact with various other devices. Note that any component, action, or functionality described above with respect toFIGS.1-10may be implemented on one or more computers configured as computer system1100ofFIG.30, according to various embodiments. In the illustrated embodiment, computer system1100includes one or more processors1110coupled to a system memory1120via an input/output (I/O) interface1130. Computer system1100further includes a network interface1140coupled to I/O interface1130, and one or more input/output devices1150, such as cursor control device1160, keyboard1170, and display(s)1180. In some cases, it is contemplated that embodiments may be implemented using a single instance of computer system1100, while in other embodiments multiple such systems, or multiple nodes making up computer system1100, may be configured to host different portions or instances of embodiments. For example, in one embodiment some elements may be implemented via one or more nodes of computer system1100that are distinct from those nodes implementing other elements.

In various embodiments, computer system1100may be a uniprocessor system including one processor1110, or a multiprocessor system including several processors1110(e.g., two, four, eight, or another suitable number). Processors1110may be any suitable processor capable of executing instructions. For example, in various embodiments processors1110may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors1110may commonly, but not necessarily, implement the same ISA.

System memory1120may be configured to store camera control program instructions1122and/or camera control data accessible by processor1110. In various embodiments, system memory1120may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions1122may be configured to implement a lens control application1124incorporating any of the functionality described above. Additionally, existing camera control data1132of memory1120may include any of the information or data structures described above. In some embodiments, program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory1120or computer system1100. While computer system1100is described as implementing the functionality of functional blocks of previous Figures, any of the functionality described herein may be implemented via such a computer system.

In one embodiment, I/O interface1130may be configured to coordinate I/O traffic between processor1110, system memory1120, and any peripheral devices in the device, including network interface1140or other peripheral interfaces, such as input/output devices1150. In some embodiments, I/O interface1130may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory1120) into a format suitable for use by another component (e.g., processor1110). In some embodiments, I/O interface1130may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface1130may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface1130, such as an interface to system memory1120, may be incorporated directly into processor1110.

Network interface1140may be configured to allow data to be exchanged between computer system1100and other devices attached to a network1185(e.g., carrier or agent devices) or between nodes of computer system1100. Network1185may in various embodiments include one or more networks including but not limited to Local Area Networks (LANs) (e.g., an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., the Internet), wireless data networks, some other electronic data network, or some combination thereof. In various embodiments, network interface1140may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.

Input/output devices1150may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or accessing data by one or more computer systems1100. Multiple input/output devices1150may be present in computer system1100or may be distributed on various nodes of computer system1100. In some embodiments, similar input/output devices may be separate from computer system1100and may interact with one or more nodes of computer system1100through a wired or wireless connection, such as over network interface1140.

As shown inFIG.11, memory1120may include program instructions1122, which may be processor-executable to implement any element or action described above. In one embodiment, the program instructions may implement the methods described above. In other embodiments, different elements and data may be included. Note that data may include any data or information described above.