Image taping in a multi-camera array

Multiple cameras are arranged in an array at a pitch, roll, and yaw that allow the cameras to have adjacent fields of view such that each camera is pointed inward relative to the array. The read window of an image sensor of each camera in a multi-camera array can be adjusted to minimize the overlap between adjacent fields of view, to maximize the correlation within the overlapping portions of the fields of view, and to correct for manufacturing and assembly tolerances. Images from cameras in a multi-camera array with adjacent fields of view can be manipulated using low-power warping and cropping techniques, and can be taped together to form a final image.

BACKGROUND

1. Technical Field

This disclosure relates to a camera array and, more specifically, to methods for capturing images using the camera array.

2. Description of the Related Art

Digital cameras are increasingly used in outdoors and sports environments. Using a camera to capture outdoors and sports environments, however, can be difficult if the camera is bulky or cannot capture the field of view desired. A user's experience with a camera can be diminished by camera bulkiness and limited camera functionality.

DETAILED DESCRIPTION

Example 2×1 and 2×2 Camera Array Configuration

A camera array configuration includes a plurality of cameras, each camera having a distinctive field of view. For example, the camera array can include a 2×1 camera array, a 2×2 camera array, or any other suitable arrangement of cameras. Each camera can have a camera housing structured to at least partially enclose the camera. Alternatively, the camera array can include a camera housing structured to enclose the plurality of cameras. Each camera can include a camera body having a camera lens structured on a front surface of the camera body, various indicators on the front of the surface of the camera body (such as LEDs, displays, and the like), various input mechanisms (such as buttons, switches, and touch-screen mechanisms), and electronics (e.g., imaging electronics, power electronics, etc.) internal to the camera body for capturing images via the camera lens and/or performing other functions. In another embodiment, the camera array includes some or all of the various indicators, various input mechanisms, and electronics and includes the plurality of cameras. A camera housing can include a lens window structured on the front surface of the camera housing and configured to substantially align with the camera lenses of the plurality of cameras, and one or more indicator windows structured on the front surface of the camera housing and configured to substantially align with the camera indicators.

FIG. 1Aillustrates a conventional multiple camera environment100, according to one embodiment. In the embodiment, the conventional camera environment100includes two cameras105A and105B. In this environment100, the camera105A is used to capture a left side (e.g., field of view108A) of a shared view115as the image110A and the camera105B is used to capture a right side (field of view108B) of the shared view115as the image110B. A portion of the field of view108A of the left camera105A and a portion of the field of view108B of the right camera105B represent a common field of view, as illustrated by the shaded portion of the shared view115. The common field of view includes a cube and a sphere. Based on the shared portions of images110A and110B, the images110A and110B can be taped together to form a single flat image of the shared view115.

Even though cameras105A and105B have a partial common field of view, any object in the common field of view may not be aligned in images captured by the left camera105A and the right camera105B. Thus, as seen in the left image110A, a part of the sphere appears to the left of the cube and, in the right image110B, a part of the sphere appears to the right of the cube. This offset120of the cube and sphere are due to parallax error. The parallax error is inherent to the conventional camera environment100because the distance between the cameras causes the position of objects in the common field of view to be different relative to each to each camera, based on the distance between the cameras. Parallax error can be further exacerbated based on a position of a read window130of an image sensor window140of an image sensor within each camera. For example, the read window130is not necessarily centered on the image sensor window140and, therefore, can result in a greater effective distance the cameras105A and105B. The position of the read window130within an image sensor window140is an issue that can arise during the manufacturing of the image sensor and can be adjusted for various purposes, as further described inFIGS. 7 and 8.

FIG. 1Billustrates a 2×1 camera array arranged according to one embodiment of the configurations described herein. In the embodiment, the 2×1 camera array150includes two cameras105C and105D. In another embodiment, as illustrated and described further in conjunction withFIG. 3, two 2×1 camera arrays150are combined to form a 2×2 camera array150that includes four cameras. In this array150, the camera105C captures a left side (e.g., field of view108C) of the shared view115and the camera105D captures a right side (e.g., field of view108D) of the shared view115. However, the left camera105C captures an image110C of the right side of the shared view115and the right camera105D captures an image110D of the left side of the shared view. The left camera105C and right camera105D have a partial common field of view, as illustrated in the shaded portion of array150. However, the partial common field of view inFIG. 1Bhas a different shape than the partial common field of view inFIG. 1A. The partial common field of view shared by cameras105C and105D includes a cube and a sphere. Based on the shared portions of images110C and110D, the images110C and110D can be taped together to form a single angled image of the shared view115.

The angled image formed by taping the images110C and110D together is different than the flat image formed by taping the images110A and110B together inFIG. 1A. The 2×1 camera array150captures images at fields of view that are not parallel, as is the case for the array100inFIG. 1A, but instead captures images at fields of view that are angled with respect to each other. The angled image is a result of the configuration of the cameras in the 2×1 camera array150as further described in conjunction withFIGS. 2 and 3. Correction of the angled image of the shared view115to simulate images captured with parallel fields of view is described further in conjunction withFIG. 9.

Unlike the environment100inFIG. 1A, the positioning of the camera within the array150ofFIG. 1Ballows both cameras105C and105D to share a portion of the other camera's field of view while minimizing parallax error of any object in the shared portion captured by the cameras105C and105D. For instance, the left image110C includes a smaller portion of the sphere that appears on the left of the cube than left image110A, and the right image110D includes a smaller portion of the sphere that appears on the right of the cube than right image110B. For example, the offset160of the left image110C can be 1-3 pixels when capturing objects within 3 meters of the 2×1 camera array150. The 2×1 camera array150minimizes distance between the cameras105A and105B, minimizing parallax error. For example, the distance between the cameras105C and105D (measured from the center of each camera lens) can be between 1 mm and 60 mm.

It should be noted that the orientation of cameras105C and105D is such that the vectors normal to each camera lens of cameras105C and105D intersect within the partial common field of view. In contrast, while the cameras105A and105B have a partial common field of view, vectors normal to each camera lens of cameras105A and105B do not intersect within the partial common field of view. In embodiments with a 2×2 camera array, vectors normal to each camera lens in the array can intersect within a field of view common to all cameras. For example, for any two cameras, vectors normal to the camera lenses of the two cameras can intersect within a field of view common to all four cameras.

FIG. 2illustrates roll, pitch, and yaw rotations of a camera in a camera array, according to one embodiment. For example, when minimizing distance between cameras in a camera array, a camera can be rotated in three dimensions: roll210, pitch220, and yaw230. By rotating a camera in these three dimensions to minimize distance between lenses of the camera with another camera, the shared view115being captured by each of a plurality of cameras can be captured at tilted, angled, or rotated fields of view. The captured images of the shared view115with the tilted, angled, and rotated fields of view can be corrected using image processing as described further in conjunction withFIG. 9.

The camera105illustrated inFIG. 2has a roll210rotation in a range of 30-50°, a yaw230rotation in a range of 60-80°, and a pitch220rotation of 60-80°. The direction of each rotation (i.e., roll210, yaw230, pitch220) of the camera105depends on the position of the camera105in the array. For example, if the camera105was in the top left of a camera array, as illustrated inFIG. 2, the camera105would have a negative roll rotation, a negative yaw rotation, and a negative pitch rotation, wherein direction is respective to the arrows illustrated inFIG. 2. If the camera105was in the top right of a camera array, the camera105would have a positive roll rotation, a positive yaw rotation, and a negative pitch rotation. If the camera105was in the bottom left of a camera array, the camera105would have a positive roll rotation, a negative yaw rotation, and a positive pitch rotation. If the camera105was in the bottom right of a camera array, the camera105would have a positive roll rotation, a positive yaw rotation, and a positive pitch rotation. The directions of rotations210,220, and230can also be based on the shape of the camera105, as further described in conjunction withFIG. 4. However, in general, the directions of the rotations210,220, and230are determined to minimize distances between the lenses of the array to a range of, for example, 1 mm-5 mm, as further described in conjunction withFIG. 3. It should be noted in other embodiments, each camera105can be oriented at a different roll, yaw, and pitch than described herein.

FIG. 3illustrates a 2×2 camera array arranged according to one embodiment of the configurations described herein. The 2×2 camera array300includes four cameras105A,105B,105C, and105D. Each camera has a roll rotation, a pitch rotation, and a yaw rotation, the rotations minimizing distance310between centers of the lenses, as indicated by the dashed lines320, to be within a range of, for example, 1 mm to 10 mm. The cameras105A-D in the illustrated 2×2 camera array300each capture an angled image of a shared view of the 2×2 camera array300. In other words, each camera captures an image of a corresponding field of view, and the corresponding field of views of the cameras overlap but are not parallel. The camera105A captures the bottom left portion of the shared view, the camera105B captures the bottom right portion of the shared view, the camera105C captures the top left portion of the shared view, and the camera105D captures the top right portion of the shared view. As used herein, “shared view” refers to each of the corresponding fields of view of the cameras105A-D of the 2×2 camera array300.

The camera array300can be adapted to be at least partially enclosed by a protective camera housing (not illustrated in the embodiment ofFIG. 3). In one embodiment, the camera array300and/or housing of the array300has a small form factor (e.g., a height of approximately 1 to 6 centimeters, a width of approximately 1 to 6 centimeters, and a depth of approximately 1 to 2 centimeters), and is lightweight (e.g., approximately 50 to 150 grams). The housing and/or camera bodies can be rigid (or substantially rigid) (e.g., plastic, metal, fiberglass, etc.) or pliable (or substantially pliable) (e.g., leather, vinyl, neoprene, etc.). In one embodiment, the housing and/or the array may be appropriately configured for use in various elements. For example, the housing may include a waterproof enclosure that protects the camera array300from water when used, for example, while surfing or scuba diving. In some embodiments, such as those described below, the camera array can300can be secured within a protective multiple camera array module, which in turn can be secured within a camera body in one or more orientations.

Portions of the housing and/or array may include exposed areas to allow a user to manipulate buttons that are associated with the camera array300functionality. Alternatively, such areas may be covered with a pliable material to allow the user to manipulate the buttons through the housing. For example, in one embodiment the top face of the housing includes an outer shutter button structured so that a shutter button of the camera array300is substantially aligned with the outer shutter button when the camera array300is secured within the housing. The shutter button of the camera array300is operationally coupled to the outer shutter button so that pressing the outer shutter button allows the user to operate the camera shutter button.

In one embodiment, the front face of the housing includes one or more lens windows structured so that the lenses of the cameras in the camera array300are substantially aligned with the lens windows when the camera array300is secured within the housing. The lens windows can be adapted for use with a conventional lens, a wide angle lens, a flat lens, or any other specialized camera lens. In this embodiment, the lens window includes a waterproof seal so as to maintain the waterproof aspect of the housing.

In one embodiment, the housing and/or array includes one or more securing structures for securing the housing and/or array to one of a variety of mounting devices. For example, various mounts include a clip-style mount or a different type of mounting structure via a different type of coupling mechanism.

In one embodiment, the housing includes an indicator window structured so that one or more camera array indicators are substantially aligned with the indicator window when the camera array300is secured within the housing. The indicator window can be any shape or size, and can be made of the same material as the remainder of the housing, or can be made of any other material, for instance a transparent or translucent material and/or a non-reflective material.

The housing can include a first housing portion and a second housing portion, according to one example embodiment. The second housing portion detachably couples with the first housing portion opposite the front face of the first housing portion. The first housing portion and second housing portion are collectively structured to enclose a camera array300within the cavity formed when the second housing portion is secured to the first housing portion in a closed position.

The camera array300is configured to capture images and video, and to store captured images and video for subsequent display or playback. The camera array300is adapted to fit within a housing, such as the housing discussed above or any other suitable housing. Each camera105A-D in the array300can be an interchangeable camera module. As illustrated, the camera array300includes a plurality of lenses configured to receive light incident upon the lenses and to direct received light onto image sensors internal to the lenses.

The camera array300can include various indicators, including LED lights and a LED display. The camera array300can also include buttons configured to allow a user of the camera array300to interact with the camera array300, to turn the camera array300on, and to otherwise configure the operating mode of the camera array300. The camera array300can also include a microphone configured to receive and record audio signals in conjunction with recording video. The camera array300can include an I/O interface. The I/O interface can be enclosed by a protective door and/or include any type or number of I/O ports or mechanisms, such as USC ports, HDMI ports, memory card slots, and the like.

The camera array300can also include a door that covers a removable battery and battery interface. The camera array300can also include an expansion pack interface configured to receive a removable expansion pack, such as a display module, an extra battery module, a wireless module, and the like. Removable expansion packs, when coupled to the camera array300, provide additional functionality to the camera array300via the expansion pack interface.

FIG. 4AandFIG. 4Billustrate a conventional multi-part camera frame405A and a single-body camera frame405B, respectively, according to one embodiment. The conventional multi-part camera frame405A includes a plurality of lenses410, an image sensor415, and auto-focus coils420. In order to enclose all the components410,415and420, the frame405A has a square profile, as illustrated inFIG. 4A. The conventional multi-part camera frame405A limits the proximity of the center of lenses of two adjacent multi-part camera frames405A.

A single-body camera frame405B can be used to improve proximity of the center of lenses of adjacent camera frames. The single-body camera frame405B (also referred to herein as a “lens stack”) includes the plurality of lenses410and the image sensor415but does not include the auto-focus coils420. In addition, the single-body camera frame405B is a single mold that holds the components410and415within the frame405B in a fixed position, set by the mold, and minimizes excess space within the frame405B. The cross-section of the frame405B is trapezoidal in shape, allowing adjacent cameras to be configured such that the proximity between the centers of the lenses of the cameras is reduced. In some embodiments, for lenses that are approximately 1 mm in diameter, the distance between the centers of the lenses of adjacent cameras is 1 mm or less.

Camera Array Block Diagrams

FIG. 5illustrates a block diagram of a multiple camera array, according to one embodiment. The array300includes four cameras500A,500B,500C, and500D, for example of cameras105A,105B,105C, and105D ofFIG. 3, and each camera includes an image sensor510, a sensor controller515, a processor520, and memory525. In another embodiment, the four cameras500A,500B,500C, and500D can have image sensors that share a common processor520, and memory525. The synchronization interface505synchronizes the four cameras500A,500B,500C, and500D to synchronously capture images. In various embodiments, the cameras500A,500B,500C, and500D can include additional, fewer, or different components for various applications. As used herein, the synchronous capture of images refers to the capture of images by two or more cameras at substantially the same time, or within a threshold period of time.

The image sensor510is a device capable of electronically capturing light incident on the image sensor510. In one embodiment, CMOS sensors are used, including transistors, photodiodes, amplifiers, analog-to-digital converters, and power supplies. In one embodiment, the image sensor510has rolling shutter functionality, and can capture light incident upon different portions of the image sensor at slightly different times. Alternatively, the image sensor510can be a CCD sensor configured to can capture the portions of the image at substantially the same time. In one embodiment, the image sensor510has an adjustable read window130. An adjustable read window130can modify the portions of an image sensor that are exposed to light and read to capture an image, or can modified the portions of an image sensor completely exposed to light that are read out to capture an image. By adjusting the read window130, the camera500A can modify when a portion of an image is captured relative to when image capture begins. For example, by shifting the read window130in a rolling shutter direction, the image sensor captures portions of the image in the read window130earlier than if the read window130was not shifted in the rolling shutter direction. Additionally, adjusting the read window130can be used to address inherent tolerance issues with the camera500A and adjust convergence point of the camera array, as further described in conjunction withFIGS. 7 and 8.

The processor520is one or more hardware devices (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and the like) that execute computer-readable instructions stored in the memory525. The processor520controls other components of the camera based on the instructions that are executed. For example, the processor520may send electronic control signals to the image sensor510or use the synchronization interface505to send data to cameras500B,500C, and500D.

The memory525is a non-transitory storage medium that can be read by the processor520. The memory525may contain volatile memory (e.g., random access memory (RAM)), non-volatile memory (e.g., a flash memory, hard disk, and the like), or a combination thereof. The memory525may store image data captured by the image sensor510and computer-readable instructions to be executed by the processor520.

The sensor controller515controls operation of the image sensor510and other functions of the camera500A. The sensor controller515can include physical and/or electronic input devices, such as exterior buttons to start recording video and/or capture a still image, a touchscreen with tap-to-focus capabilities, and a dial/button combination for navigating a menu hierarchy of the camera500A. In addition, the sensor controller15may include remote user input devices, such as remote controls that wirelessly communicate with the cameras500A-D. The image sensor510may function independently of the sensor controller515. For example, a slave camera in a master-slave pairing can receive a signal from the master camera to capture an image through the synchronization interface505.

The synchronization interface505sends data to and receives data from cameras500A,500B,500C, and500D, or an external computing system. In particular, the synchronization interface505may send or receive commands to cameras500A,500B,500C, and500D for simultaneously capturing an image and/or calibrating synchronization with the cameras500A,500B,500C, and500D (e.g., sending or receiving a synchronization pulse). In the illustrated embodiment ofFIG. 5, there is one synchronization interface505controlling the cameras500A,500B,500C, and500D. In another embodiment, there can be a plurality of synchronization interfaces505controlling the cameras500A-D, for instance, one synchronization interface505per camera.

FIG. 6illustrates a block diagram of a synchronization interface for a multiple camera array, according to one embodiment. The synchronization interface505includes an image store605, a synchronization store610, a capture controller615, a pixel shift determination module620, a time lag determination module625, and an image capture module630. Alternate embodiments may have one or more additional, omitted, or alternative modules configured to perform similar functionality. It should be noted that in other embodiments, the modules described herein can be implemented in hardware, firmware, or a combination of hardware, firmware, and software. In addition, in some embodiments, a first camera in a plurality of cameras includes the components illustrated inFIG. 6, while the other cameras in the plurality of cameras do not necessarily include the components ofFIG. 6and instead merely synchronously capture an image in response to a signal from the first camera. As used herein, a “plurality of images” refers to a plurality of images captured synchronously by the plurality of cameras, each camera capturing a portion of a field of view shared with two adjacent cameras. Alternatively or additionally, an external computing device processes image data captured by the camera array.

The image store605is configured to store a plurality of images synchronously captured by each of a plurality of cameras, such as the cameras500A-D ofFIG. 5. The synchronization store610is configured to store received camera synchronization data. Examples of synchronization data include time lags between cameras due to network lag or internal component lag (e.g., lag from the synchronization interface505, the processor520, the sensor controller515, and the like). The synchronization store610is configured to store calibration settings, such as read window shift information and a calibrated time lag for initiating image capture, for use in calibrating the cameras in a camera array based on, for example, camera synchronization data.

The capture controller615controls image capture by the image sensor510. In one embodiment, the capture controller615applies a calibration correction to synchronize image capture with one or more additional cameras, for instance based on synchronization or calibration data stored in the synchronization store610. The calibration correction may include a read window shift by a determined number of pixels, as determined by the pixel shift determination module620. The calibration correction may include, alternatively or additionally, a time lag for one of the cameras in the array to delay relative to the other camera of the array before beginning image capture, as determined by the time lag determination module625.

The pixel shift determination module620identifies a pixel shift between an image captured by a first camera500A, an image captured by a second camera500B, an image captured by a third camera500C, and an image captured by a fourth camera500D. This pixel shift indicates spatial misalignment between the image sensors of the cameras500. In one embodiment, the pixel shift determination module620determines a pixel shift in a rolling shutter direction due to a misalignment between the image sensors along the rolling shutter direction. The capture controller615can use the determined pixel shift to correct the misalignment between the image sensors.

The time lag determination module625determines a time lag between the capture of an image row by a first camera500A, a corresponding image row of a second camera500B, a corresponding image row of a third camera500C, and a corresponding image row of a fourth camera500D. The time lag determination module625can determine a time lag based on a pixel shift received from the pixel shift determination module620. Using the determined time lag, tlag, the capture controller615synchronizes the plurality of cameras by delaying image capture of a first of the plurality by the time lag relative to a second, third and fourth of the plurality. In one embodiment, an image sensor has an associated row time, trow, which represents an elapsed time between exposing a first pixel row and a second, subsequent pixel row. If images taken by a plurality of cameras are determined to have a pixel shift of n pixels, then the time lag tlagrequired to correct the pixel shift can be determined using the following equation:
tlag=trow×n

In one embodiment, calibrating image capture between cameras in a plurality of cameras involves synchronously capturing images with the plurality of cameras, determining a pixel shift between the captured images, and applying a determined correction iteratively until the determined pixel shift is less than a pre-determined pixel shift threshold. The calibration process may be initiated when cameras are powered on or paired, when the cameras are manufactured, when the camera array is assembled, or in response to a manual initiation of the calibration process by a user of the camera array. A master camera system can initiate the calibration process after an amount of time elapses that is greater than or equal to a pre-determined threshold since a previous calibration. In an embodiment with additional cameras, additional calibrations can be performed among other cameras in response to the calibration of the master camera.

The image capture module630processes captured images. In an alternative embodiment not described further herein, the captured images are processed outside of the synchronization interface505, for instance by a system external to the camera array. The image capture module630includes a tolerance correction module632, a convergence adjustment module634, and an image processing module636. Alternative embodiments may have one or more additional, omitted, or alternative modules configured to perform similar functionality.

The tolerance correction module632shifts read windows of the plurality of cameras500to correct for tolerances in each camera in the plurality. For example, each camera can have slight differences in read window location due to manufacturing discrepancies and product tolerances of the cameras. In addition, image sensors of the plurality of cameras can have varying read window locations. These variations from camera to camera can be due to discrepancies between locations and orientations of the image sensor within each camera. For example, if the image sensor is a complementary metal-oxide-semiconductor (CMOS) sensor, the location of the read window for each CMOS sensor can shift due to the sensitivity of pixel sensors used in each CMOS sensor. Shifted read window locations in CMOS sensors of cameras in the camera array can shift the field of views of cameras, as shown in the field of views of cameras in the camera array700inFIG. 7A. This can result in a shifted shared field of view (e.g., shaded portion unaligned with the field of views of the cameras in the camera array700). For example, the tolerance correction module632can recognize a shift in read window locations based on outputs of pixel sensors in the read windows. The outputs of image sensors in a first region of a first camera and a second region of a second camera, where the first region and the second region overlap, can be compared, and the location of the read window of an image sensor of the first camera, the second camera, or both can be shifted based on the comparison. For instance, upon comparing overlapping image sensor regions, if a determination is made that a first of the regions is offset from a second of the regions by a determined number of pixels, the first region or the second region can be shifted by the number of pixels (for instance, by adjusting a location of the read window on an image sensor by the number of pixels) to align the regions. The process of correcting for tolerance of the cameras500is further described below in conjunction withFIG. 7.

The convergence adjustment module634can dynamically adjust a read window in an image sensor based on image data captured by the camera array. For example, if the camera array is capturing an image of an object in a foreground, the convergence point of the plurality of cameras in the camera array can be adjusted closer to the camera array and, if capturing an object in a background, the convergence point of the plurality of cameras in the camera array can be adjusted farther away from the camera array. In another example, the convergence point of the camera array is only adjusted by the convergence adjustment module634if the depth of the object in the field of view of the camera array exceeds a threshold distance. The convergence adjustment can be done manually by a user of the camera array or automatically by the convergence adjustment module634based on image processing algorithms measuring a depth of an object in the field of view of the camera array. The process of adjusting convergence of the plurality of cameras500is further described in conjunction withFIG. 8.

The image processing module636adjusts images captured by the camera array to compensate for the angled fields of view of the cameras of the camera array. The images are adjusted using, for example warps, transformations, crops, or any other suitable image enhancement, restoration, and/or compression techniques. One or more of the images can be adjusted individually, or all of the images can be adjusted substantially synchronously or sequentially. For example, prior to and during the taping of the images to produce a simulated flat image, as further described below in conjunction withFIG. 9.

Read Window Adjustment

FIG. 7Aillustrates a read window adjustment for each image sensor in a multiple camera array for tolerance compensation, according to one embodiment. For example, illustrated is an imperfectly aligned 2×1 camera array700illustrated in the shaded area misaligned with the lines representing the common field of view of the cameras in the camera array700. A portion of each of the fields of view of cameras705A and705B is common to both fields of view, and includes a cube. However, images710A and710B captured by the cameras705A and705B, respectively, illustrate that the cube is not aligned on the x-axis or y-axis of the read windows of the image sensors in the cameras705.

In the illustrated example, the images710A and710B display the corner of the cube at (x,y) coordinates (3,4) and (12,6), respectively, where the read window has a height of 12 pixels and 20 pixels and the cube a width of 3 pixels. Thus, the cube has different heights within the read windows, and has different locations along the x-axis within the read windows. The read window of the image sensor of each camera in the plurality of cameras705A and705B can be adjusted to display the center of the cube at the same height in the y-axis and distance from the respective edges in the x-axis. Images710C and710D illustrate adjusted read windows, showing the corner of the cube displayed at (4,5) and (13,5), respectively (a same height and distance from the image edge). Detection of the cube can be an automatic process based on an object detection algorithm or assisted by a user of the camera array700and the user's input.

FIG. 7Billustrates unaligned read windows130within image sensor windows140of a multiple camera array, according to another embodiment. Images captured with unaligned read windows130in a multiple camera array have can significant decorrelation within the overlapping portions of the captured images. Aligning the read windows130within the image sensor capture windows140of a multiple camera array can increase the correlation between overlapping portions of captured images, thus beneficially improving performance when stitching together adjacent images as described further herein below.

To align the read windows130within a multiple camera array, an object of interest can be identified. For instance, if a multiple camera array is used to capture a set of images (one per camera) of a garden, a garden flower within the images can be selected. A set of correlation coefficients representative of an amount of decorrelation within the overlapping portions of the captured set of images is determined. The set of correlation coefficients can be weighted such that correlation coefficients associated with the identified object of interest are weighted more heavily than other correlation coefficients. In some embodiments, correlation coefficients associated with a center region representative of an overlap between each of the images in the set of images are weighted more heavily. In some embodiments, the further away from the object of interest or the center region a correlation coefficient is, the less it is weighted.

The read windows130(such as read windows A, B, C, and D ofFIG. 7B) are adjusted (moved within an image sensor capture window140) such that the weighted set of correlation coefficients is lowest. In other words, the read windows130are re-located within the image sensor capture windows140, a set of correlation coefficients is determined for a set of images captured using the adjusted read windows130of a multiple camera array, and if the resulting weighted set of correlation coefficients is representative of less decorrelation, the adjusted read windows130are selected over the previous read windows130. In some embodiments, read windows130can be adjusted using a lookup table or function describing a relationship between correlation coefficients and read window location/adjustment. In some embodiments, instead of using a set of correlation coefficients, other representations of correlation within overlapping read window portions can be used such that a minimum or near-minimum entropy can be determined in response to the iterative adjusting of read windows as described herein

This process can be repeated a number of times, for instance, until the captured images represent a below threshold level of decorrelation.FIG. 7Cillustrates aligned read windows130A′, B′, C′, and D′ within image sensor capture windows140of a multiple camera array, according to one embodiment. In some embodiments, read windows130are adjusted until the line of sight of each lens in a multiple camera array converges at a distance d determined to result in an optimal level of correlation between a captured set of images.FIG. 7Dillustrates a distance of optimal correlation between image sensors of a multiple camera array700.

FIGS. 8A-8Cillustrates a read window adjustment for each image sensor in a multiple camera array for convergence point adjustment, according to one embodiment. A plurality of read windows130of image sensor windows140in a camera array800and a corresponding convergence point820based on a distance825of a shift of the read windows are illustrated inFIGS. 8A-8C. As illustrated inFIG. 8A, the camera array800includes a left camera805A and a right camera805B that capture an image810A of the right side of a shared view115and an image810B of the left side of the shared view115. The substantial center815between the cameras805results in a substantial center of a portion of shared fields of view of the captured images810A and810B, as shown in the vertical dotted lines815. Both captured images810A and810B capture a cube seen by both cameras805A and805B within the shared view115.

When capturing the shared view115, the distance between the center of the lenses of the cameras805A and805B affect the convergence point820of the lenses of the cameras805A and805B. This distance between the center of the lenses can be increased and decreased by shifting the read windows130of the image sensor windows140within the cameras805A and805B, as illustrated inFIG. 8A-8C. Thus, the distance825A of the portion of the shared fields of view between the cameras805A and805B in the read windows130affects the distance between the center of the lenses of the cameras805A and805B and results in a convergence point at820A.

For example, as the distance825increases, as seen in distance825B inFIG. 8B, the distance between the center of the lenses of the cameras805A and805B decreases, because the distance between the read windows decreases, and results in a convergence point at820B behind the object, farther from the camera array800. As the distance825decreases, as seen in distance825C inFIG. 8C, the distance between the center of the lenses of the cameras805A and805B increases, because the distance between the read windows increases, and results in a convergence Point at820C in front of the object, closer to the camera array800.

In various embodiments, the distance between the read windows can increase or decrease from shifting one or both of the read windows of the image sensors of the cameras805A and805B. The shift of the read windows can be done automatically based on object detection algorithms or by a manual input of a user using the camera array800. For example, a user can input a setting or a mode (e.g., landscape, portrait, macro, sports, night, movie) and based on the input the read windows are shifted to better capture an image of that type of mode.

Image Processing & Taping

FIG. 9Aillustrates a set of images captured by cameras in a 2×2 camera array, according to one embodiment. For example, the camera array300ofFIG. 3collectively captured a shared view115, the camera105A captured image910C, the camera105B captured image910D, the camera105C captured image910A, and the camera105D captured image910B. For purposes of discussion, the image processing steps discussed herein will be discussed in the context of the image910A captured by the camera105C. However, the image processing steps are also performed in the same order on the other images910B,910C, and910D. In another embodiment, the image processing steps applied to the images can be different than the image processing steps described herein, or can be performed in a different order on the other images910B,910C, and910D.

The images captured by the camera array can vary in distortion and warp based on the camera in the camera array or the position of the camera in the camera array (e.g., roll, pitch, yaw, etc.). Thus, as seen in image910A, if the camera is a fish eye camera, the captured image910A has a fish eye distortion. In addition, the portions of the shared field of view of image910A are angled at a different orientation than adjacent images910B and910C, as each image was captured at different angled fields of view. Since the images910A,910B,910C, and910D are of a shared view and each image shares a portion of a shared field of view920with an adjacent image, common objects, such as objects930, are visible in the portions of the shared field of view920. For example, the common object930AB between910A and910B is in the portion of the shared field of view920AB, the common object930AC between910A and910C is in the portion of the shared field of view920AC, the common object930BD between910B and910D is in the portion of the shared field of view920BD, and the common object930CD between910C and910D is in the portion of the shared field of view920CD. Thus, each image has a first portion representative of an overlapping field of view with a corresponding portion of a horizontally adjacent image and a second portion representative of an overlapping field of view with a corresponding portion of a vertically adjacent image. In the example shown, each common object930is warped differently in each adjacent image due to the fish eye distortion.

FIG. 9Billustrates the captured images ofFIG. 9Aaligned based on the overlapping portions, according to one embodiment. Following the example from before, the common objects930are supposed to be vertical and horizontal straight lines but are warped due to the fish eye distortion. Therefore, since the common objects930are supposed to be straight or, in general, since the shapes of the common objects930are known, the captured images ofFIG. 9A(illustrated in the dotted line around image910A) can be processed, as illustrated inFIG. 9B, so that common objects930are aligned between adjacent images (in other words, so that common objects930have the correct orientation and shape between adjacent images). The alignment can be performed by a warp function or any other suitable image processing algorithm that stretches the image in a manner that results in the common object930being aligned in shape and orientation between adjacent images. In one embodiment, the warp only straightens the common objects930or corrects distortion of the common objects930and regions near the common objects930while leaving the outer edges farther from the common objects in their distorted state. In the embodiment shown here, the warp is performed not locally but on the entire image910, resulting in correction of the common objects930and warping of the outer edges of the images910.

FIG. 9Cillustrates the aligned images ofFIG. 9Bcropped to remove excess portions not horizontally or vertically aligned with corresponding adjacent images, according to one embodiment. The aligned images (illustrated in the dotted line around image910A) are cropped (such as the image910A) along the x-axis and y-axis of the image at the outer edges of the four images910A,910B,910C, and910D, resulting in a cropped image with at least 2 straight edges (as illustrated in each of the images inFIG. 9C). In other words, for each image, portions of the image that are not both horizontally and vertically aligned with portions of the image representative of shared fields of view with adjacent images are cropped.

FIG. 9Dillustrates the cropped images ofFIG. 9Cwarped to correct for distortions based on the alignment, according to one embodiment. For example, when the aligned images are warped to straighten out the common objects930, the outer edges of the aligned images are also warped. The magnitude of the warping to correct for distortions here is based on a distance of the portion to the first portion of the image and the second portion of the image, wherein the magnitude of the second warping of a first portion of the image representative of a shared field of view with an adjacent image and a second portion of the image representative of a shared field of view with an adjacent image is substantially 0. In an embodiment where only the regions near the common objects930are warped, there is no need for correcting distortions of the cropped images ofFIG. 9C.

In general, the distortions corrected are the distortions caused by the alignment step fromFIG. 9A to 9B. Therefore, if there are no distortions to correct after alignment, this step is not required. Following the example illustrated, however, the outer edges of the aligned images are warped during alignment and, therefore, the distortions of the outer edges are corrected here using warp techniques and other transforms. In general, any image processing algorithm that performs the function of correcting for distortion can be performed here. The image processing algorithms performed here, however, may not affect the common objects930. In this example, the common object930is a line and, therefore, the image processing algorithms performed here to correct for distortions do not warp the common objects930and the common objects930are still linear in shape after the images910are corrected for distortions.

FIG. 9Eillustrates the corrected images ofFIG. 9Dcropped to remove excess portions and overlapping portions, according to one embodiment. The excess portions on the outer edges of the images910are removed as well as the overlapping portions of shared fields of view. Then, a final image of the shared view being captured by the camera array is generated by taping each cropped image to a horizontally adjacent cropped image and a vertically adjacent cropped image. In other embodiments, the overlapping portions of the corrected images ofFIG. 9Dare substantially reduced and each image is concatenated with each corresponding horizontally and vertically adjacent cropped image.

Additional Embodiments

FIG. 10Aillustrates a distance between two lens modules in a multiple camera array, according to one embodiment. The lens modules1000A and1000B have some distance D between the centers of the lenses. The distance between the lenses results in a parallax error, which must be corrected for when stitching together images captured in a multiple camera array. Parallax error is introduced even when the lens modules are placed very close together, for instance 5 mm or less.

To help reduce or eliminate parallax error, a single lens can be used for multiple cameras in a multiple camera array. In some embodiments, the common lens is a ball lens.FIG. 10Billustrates a ball lens for use in a multiple camera array, according to one embodiment. The ball lens1002is used for both lens module1000A and1000B. It should be noted that although only two lenses are illustrated in the embodiment ofFIG. 10B, a common lens (such as the ball lens1002) can be used for any or all of the cameras in a multiple camera array, such as all 4 cameras in a 2×2 camera array.

The optical paths of light through the ball lens1002and upon each lens module1000intersects within the ball lens1002. Accordingly, the distance D between the centers of the lenses is reduced to zero, effectively eliminating parallax error. It should be noted that in some embodiments, a ball lens1002introduces distortion to images captured using a ball lens1002; in such embodiments, an additional warp function can be applied to each image captured using the ball lens1002prior to stitching the images together to reduce the effects of the distortion introduced by the ball lens1002.

FIG. 11Aillustrates a camera body1100configured to receive and secure a multiple camera array module1110in a plurality of configurations, according to one embodiment. The camera body1100can include camera circuitry, interfaces, and the like (not shown) configured to provide a camera interface to a user of the camera body1100. The camera body1100includes a cavity1105, for instance on the front surface of the camera body1100, configured to receive a multiple camera array module1110in any of a number of configurations.

FIG. 11Billustrates a multiple camera array module1110for insertion within a camera body1100in a plurality of configurations, according to one embodiment. The multiple camera array module1110(e.g., 2×2 camera array) includes 4 cameras1115A-D coupled together to form a substantially rectangle shape, with two protrusions extending away from the camera array module1110along the midpoints of two adjacent sides of the multiple camera module1110. In other embodiments, the camera array module1110can be in a substantially circular shape with similar protrusions. The cavity1105of the camera body1100can include reciprocal cavities arranged to accommodate the protrusions of the multiple camera array module1110in any number of configurations. In another embodiment, the camera array module1110can include flexible material where the protrusions are and the cavity1105of the camera body can include protrusions instead of reciprocal cavities. Thus, the protrusions of the cavity1105can snap into the flexible material present in the camera array module1110.

In the embodiment ofFIG. 11A, the cavity1105within the front face of the camera body1100can receive the multiple camera module1110in the position illustrated inFIG. 11B, and can further receive the multiple camera array module1110in a position rotated 90 degrees clockwise. Such a configuration beneficially allows a user to capture images in a first orientation of the camera body1100, and to remove, rotate, and re-insert the multiple camera array module1110into the camera body1100in a second orientation. In an alternative embodiment, the camera body1100can be rotated while in the camera body1100. For example, the cavity1105can include a reciprocal cavity path carved out inside the camera body1100along the edges of the cavity1105. Then, when the camera array module1110is inserted in the cavity1105, the camera array module1110can be rotated when the protrusions of the module1110align with the reciprocal cavity path inside the camera body1100. Although a particular embodiment is illustrated inFIGS. 11A and 11B, in practice any configuration of camera body1100reciprocating cavity1105and multiple camera array module1110can be used according to the principles described herein.

FIG. 12Aillustrates a camera strap1200including a battery system and an associated electrical interface, according to one embodiment. In the embodiment ofFIG. 12A, a camera strap1200includes one or more batteries (e.g.,1210A,1210B,1210C, and1210D) electrically coupled to and configured to provide power to a camera system coupled to the strap by strap wiring1205within the strap1200. For example, the camera strap1200can include battery interfaces1215A,1215B, and1215C that can each receive a battery1210and electrically couple the battery1210with the other batteries1200through the wiring1205, as shown inFIG. 12B.

FIG. 12Billustrates the camera strap1200ofFIG. 12A, according to one embodiment. The strap1200includes multiple layers: two outer protective layers1200A and1200D, a battery layer1200B configured to provide power to a battery, and a circuit layer1200C including one or more circuits (not shown) configured to perform various functions, such as a circuit configured to identify an amount of power stored by the battery, an amount of power available to the camera, and the like. The camera strap1200can be coupled to the camera array through a connection interface aligned with the wiring1205or with the battery interfaces1215. In other words, the camera strap can secure a camera or a multiple camera array to a user, and the batteries1210can provide power to the secure camera or multiple camera array via the wiring1205.

FIG. 13illustrates a grip system for a camera system, according to one embodiment. The rear view1300A of the camera system illustrates the rear of the camera. The top view1300B illustrates a grip pattern on one side of the top of the camera system. The grip pattern illustrated within the embodiment ofFIG. 13includes asymmetrical sawtooth grip/friction protrusions angled in an obtuse relation to the body in the direction of anticipated grip slippage by a user of the camera system. The grip patterns can be less than 0.5 mm in depth, and within 0.25 mm apart in some embodiments. The front view1300C illustrates two grip areas on one side of the front of the camera. Each grip area includes sawtooth protrusions angled in different directions, and extending various lengths across the front of the camera system. The different angles of the sawtooth protrusions account for different directions in expected grip slippage; for instance, the top grip area can be angled to accommodate grip slippage by a front of an index finger, and the bottom grip area can be angled to accommodate grip slippage by a side of a middle finger. The bottom view1300D illustrates a grip pattern on one side of the bottom of the camera system, and the side view1300E illustrates a grip pattern on one portion of the side of the camera system. The grip system illustrated in the embodiment ofFIG. 13beneficially increases the ability of a user to grip a camera, for instance in inclement conditions such as camera operation in cold and wet conditions. The position of the grip pattern on the camera system also beneficially guides a user's hand to a portion of the camera that does not interfere with camera buttons, interfaces, and the like, and does not obfuscate the camera lens.

FIG. 14Aillustrates adjacent lens stacks in a multiple camera array, according to one embodiment. Each lens stack, lens stack1400A and lens stack1400B, includes a stack housing1402containing one or more lenses1404. Each stack housing1402is shaped such that walls or outer surfaces of the housing converge from the rear of each lens stack1400to the front. For example, the outer layer of the stack housing1402can be less than 5 mm from the inner components including the lenses1404. Accordingly, the lenses1404in each lens stack1400have progressively smaller diameters from the rear of each lens stack1400to the front. Each lens1404in the embodiment ofFIG. 14Acan be a disc lens, a flat lens, or the like. It should be noted that although certain lenses are referred to herein as “disc lenses” or “flat lenses”, it should be noted that in practice such lenses can have curved front and/or rear faces, but that the diameter of such lenses is generally larger than the front-to-rear thickness of such lenses. The parallax error between images caught using the lens stacks1400A and1400B is dependent on the distance A between the center of each forward-most lens of lens stack1400A and1400B. As the distance A decreases, the parallax error in images captured by the lens stacks1400A and1400B decreases.

FIG. 14Billustrates adjacent lens stacks1410including cone lenses1414in a multiple camera array, according to one embodiment. Each lens stack1410A and1410B includes a plurality of lenses: one or more disc lenses1412, and one forward-most cone lens1414. As used herein, a cone lens1414refers to a lens with a circular cross-section and a variable diameter, such that a diameter of a cross-section at the rear of the cone lens1414(the side facing the rear side of the lens stack) is larger than a diameter of a cross-section at the front of the lens (the side facing the front side of the lens stack). It should be noted that although the cone lenses1414ofFIG. 14Bare shown with curved front and rear faces, in practice these faces may be flat, or may be curved to a much lesser extent than as shown inFIG. 14B.

The use of cone lenses1414as forward-most lenses in the lens stacks1410in the multiple camera array ofFIG. 14Ballows the housing1402of each lens stack1410to extend further forward in comparison to the lens stacks1400ofFIG. 14A. This in turn reduces the distance A between the centers of the forward-most lenses, which in turn reduces the parallax error in images captured by the multiple camera array. It should be noted that although only two lens stacks1410are illustrated in the embodiment ofFIG. 14B, in practice, multiple camera arrays can include any number of lens stacks, each including a cone lens as the forward-most lens within the lens stack.

It should be noted that the multiple camera arrays described herein can be configured to couple to and act as a remote control for any device configured to wirelessly communicate with the multiple camera array device. For instance, the multiple camera array can be configured to act as a remote control for another camera, a music/video playback device, a storage device, a wireless communication device, a telemetry monitoring device, a calendar control or display device, a slideshow control device, or any other wireless device.

Additional Configuration Considerations

Throughout this specification, some embodiments have used the expression “coupled” along with its derivatives. The term “coupled” as used herein is not necessarily limited to two or more elements being in direct physical or electrical contact. Rather, the term “coupled” may also encompass two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other, or are structured to provide a thermal conduction path between the elements.