Parallel image processing using multiple processors

An electronic device for parallel image processing using multiple processors is disclosed. The electronic device includes multiple image sensors for providing image data. The electronic device also includes multiple processors for processing segmented image data to produce processed segmented image data. Each processor is dedicated to one of the image sensors. A multiple processor interface is also included. The multiple processor interface maps the image data to the processors, segments the image data to produce the segmented image data and synchronizes the segmented image data to processor clock rates.

TECHNICAL FIELD

The present disclosure relates generally to electronic devices. More specifically, the present disclosure relates to parallel image processing using multiple processors.

BACKGROUND

In the last several decades, the use of electronic devices has become common. In particular, advances in electronic technology have reduced the cost of increasingly complex and useful electronic devices. Cost reduction and consumer demand have proliferated the use of electronic devices such that they are practically ubiquitous in modern society. As the use of electronic devices has expanded, so has the demand for new and improved features of electronic devices. More specifically, electronic devices that perform functions faster, more efficiently or with higher quality are often sought after.

Some functions may be data intensive (e.g., image processing), requiring a large amount of processing resources to complete in a desirable amount of time. However, electronic devices such as televisions, smart phones, computers, cameras and music players have processing constraints that determine how quickly they can perform certain functions. Furthermore, some of these electronic devices such as mobile telephones or digital cameras are limited by the amount of power they have stored in a battery. It may be difficult to economically design and manufacture an electronic device that improves processing efficiency and reduces power consumption while providing higher-quality data-intensive functions.

As can be observed from the foregoing discussion, improving the processing capability and/or efficiency of electronic devices may be beneficial. Systems and methods that improve the capability and/or efficiency of electronic devices are disclosed herein.

SUMMARY

An electronic device for parallel image processing using multiple processors is disclosed. The electronic device includes a plurality of image sensors for providing image data and a plurality of processors for processing segmented image data to produce processed segmented image data. Each processor of the plurality of processors is dedicated to one of the plurality of image sensors. The electronic device also includes a multiple processor interface. The multiple processor interface maps the image data to the plurality of processors, segments the image data to produce the segmented image data and synchronizes the segmented image data to clock rates of the plurality of processors.

The electronic device may also include a combining module for combining the processed segmented image data to produce a processed image. At least one processor of the plurality of processors may process segmented image data from at least one image sensor that the at least one processor is not dedicated to. The multiple processor interface may dynamically adds processors for image data processing from the plurality of processors. Dynamically adding processors for image data processing may be based on a workload.

The multiple processor interface may determine whether it is beneficial to segment the image data. Each processor may process image data only from an image sensor that each processor is dedicated to if the multiple processor interface determines that it is not beneficial to segment the image data. The multiple processor interface may determine that it is beneficial to segment the image data if the image data can be processed more rapidly if it is segmented. The multiple processor interface may determine that it is beneficial to segment the image data if at least one of the plurality of processors is individually incapable of processing the image data. The multiple processor interface may determine that it is beneficial to segment the image data if the image data can be processed using less power if it is segmented.

The multiple processor interface may determine a mapping to map the image data to the plurality of processors. The multiple processor interface may determine a segmentation of the image data. At least one of the plurality of processors may have a different capability from another of the plurality of processors. At least one of the plurality of image sensors may have a different capability from another of the plurality of image sensors.

The plurality of processors may be individually incapable of processing the image data. Segments of the segmented image data may overlap. The plurality of processors may process the segmented image data in real time and in parallel. The electronic device may be a wireless communication device. The plurality of processors may be image signal processors (ISPs). The electronic device may stream the image data such that the image data is not stored before processing. The combining module may include an output buffer. The combining module may include an output interface and an output buffer.

A method for parallel image processing using multiple processors is also disclosed. The method includes providing image data to an electronic device using a plurality of image sensors and mapping the image data to a plurality of processors. Each processor is dedicated to one of the plurality of image sensors. The method also includes segmenting the image data to produce segmented image data, synchronizing the segmented image data to clock rates of the plurality of processors and processing the segmented image data to produce processed segmented image data.

A computer-program product for parallel image processing using multiple processors is also disclosed. The computer-program product includes instructions on a non-transitory computer-readable medium. The instructions include code for providing image data using a plurality of image sensors and code for mapping the image data to the plurality of processors. Each processor is dedicated to one of the plurality of image sensors. The instructions further include code for segmenting the image data to produce segmented image data, code for synchronizing the segmented image data to clock rates of the plurality of processors and code for processing the segmented image data to produce processed segmented image data.

An apparatus for parallel image processing using multiple processors is also disclosed. The apparatus includes means for providing image data using a plurality of image sensors and means for mapping the image data to the plurality of processors. Each processor is dedicated to one of the plurality of image sensors. The apparatus further includes means for segmenting the image data to produce segmented image data, means for synchronizing the segmented image data to clock rates of the plurality of processors and means for processing the segmented image data to produce processed segmented image data.

DETAILED DESCRIPTION

As discussed above, improved capabilities for electronic devices are beneficial and desirable. Specifically, image sensor modules with support for higher Frames Per Second (FPS) processing and/or larger megapixel (MP) sensors are beneficial. For example, larger MP sensors are capable of capturing images with higher resolution or finer detail. Furthermore, higher FPS support in an electronic device enables the capture of smoother video. However, larger MP sensors and/or higher FPS support may require higher processing throughput. For example, the processing throughput for Image Signal Processors (ISPs) may need to increase substantially to support a 30 FPS output rate for 12 MP sensors.

Designing an image signal processor (ISP) with a substantial increase in processing throughput may be time consuming and very costly. The systems and methods disclosed herein, however, allow the combination of two or more “off-the-shelf” image signal processor (ISP) modules to process in parallel in order to achieve the desired throughput. The systems and methods disclosed herein may be especially beneficial since two or more image signal processor (ISP) modules may already be used in some electronic devices. Some of these electronic devices may include two cameras with an image signal processor (ISP) for each camera used for stereo (e.g., for 3-dimensional (3D)) video or for video telephony applications where a main camera and a secondary camera are included. The systems and methods disclosed herein can be used to combine these different camera modules to achieve high processing throughput. That is, an electronic device implementing the systems and methods disclosed herein may dynamically and arbitrarily partition image data for parallel processing. One benefit of this approach is that existing image signal processor (ISP) cores can be used to achieve desired performance without having to re-design a higher throughput pipeline. In one configuration, for example, the systems and methods disclosed herein may be applied such that N processors may be used to process the output of M sensors where M<N.

It should be noted that the terms “couple,” “coupling,” “coupled” or other variations of the word couple as used herein may indicate either an indirect connection or a direct connection. For example, if a first component is “coupled” to a second component, the first component may be either indirectly connected to the second component or directly connected to the second component. As used herein, the term “plurality” denotes two or more. For example, a plurality of components indicates two or more components.

Various configurations are now described with reference to the Figures, where like reference numbers may indicate functionally similar elements. The systems and methods as generally described and illustrated in the Figures herein could be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of several configurations, as represented in the Figures, is not intended to limit scope, as claimed, but is merely representative of the systems and methods.

FIG. 1is a block diagram illustrating one configuration of an electronic device102in which systems and methods for parallel image processing using multiple processors may be implemented. The electronic device102includes one or more image sensors104a-nand two or more processors120a-n. The electronic device102may include the same or a different number of image sensors104a-nand processors120a-n. Examples of image sensors104a-ninclude Charge-Coupled Device (CCD) sensors and Complementary Metal Oxide Semiconductor (CMOS) sensors. Examples of processors120a-ninclude dedicated image signal processors (ISPs) and software implementations of processors. For example, processor A120ais dedicated to image sensor A104a, processor B120bis dedicated to image sensor B104band so on. That is, image sensor A104aand processor A120ashare sensor-processor dedication A106a, image sensor B104band processor B120bshare sensor-processor dedication B106band so on up to sensor-processor dedication N106nshared by image sensor N104nand processor N120n. Sensor-processor dedication106a-nindicates that when the present systems and methods are not being used, each processor120a-nonly processes image data from a particular image sensor104a-nor that each processor120a-nwas designed to process image data from a particular image sensor104a-n. For example, image sensor A104aand processor A120amay be manufactured and/or packaged as a single module for use in an electronic device102. This combined image sensor104/processor120module may be an “off-the-shelf” package, where one or more processors120a-nare dedicated106to one or more image sensors104a-n.

As used herein, a “dedicated processor” or a processor with “sensor-processor dedication” may indicate that when the systems and methods herein are not used, the processor only processes image data from a particular image sensor. This processor dedication may include different types of dedication. Inclusive dedication means that a particular processor processes data only from one image sensor, though one or more additional processors may be dedicated to and process image data from that image sensor. These inclusively dedicated processors will only process data from that image sensor when the systems and methods disclosed herein are not used. For example, two processors may each be dedicated to a single image sensor, each processing data from the single image sensor when the systems and methods herein are not used. For instance, dual processors may be specifically designed to process data only from one particular image sensor. Exclusive dedication means that a processor processes only image data from a particular image sensor and is the only processor that processes data from that particular image sensor when the systems and methods herein are not used. As used herein, the terms “dedicated processor,” “sensor-processor dedication” and other variations disclose both inclusive dedication and exclusive dedication. In one configuration, the one or more processors120are each inclusively dedicated to an image sensor104. In another configuration, the one or more processors120are each exclusively dedicated to an image sensor104. The systems and methods disclosed herein may be applied to inclusively dedicated processors, exclusively dedicated processors and/or both.

The processors120a-nperform image processing operations. Examples of image processing operations include cropping, scaling (e.g., to a different resolution), image format conversion, color interpolation, color processing, image filtering (e.g., spatial image filtering), lens artifact or defect correction, etc. Each of the processors120a-nmay have different processing capabilities. For example, processor A120amay process image data at a higher clock rate than processor B120b. Similarly, each of the image sensors104a-nmay have different capabilities. For example, image sensor A104amay provide image data at a higher frame rate and/or resolution than image sensor B104b.

According to the systems and methods disclosed herein, a multiple processor interface108may combine the processing capabilities of two or more processors120a-neven though the two or more processors120a-nare dedicated106a-nto specific image sensors104a-n. For example, the multiple processor interface108may use interfacing118to map image data from one or more image sensors104a-nto one or more processors120a-n. In some configurations, the multiple processor interface108and interfacing118are a single module. In other configurations, the multiple processor interface108and interfacing118are separate. In one configuration, the multiple processor interface108and/or interfacing118are implemented as hardware components (e.g., switches and/or multiplexers to map image data from one or more image sensors104a-nto one or more processors120a-n). In another configuration, the multiple processor interface108and/or interfacing118are implemented as software (e.g., for addressing or mapping image data from one or more image sensors104a-nto one or more processors120a-n). In yet another configuration, the multiple processor interface108and/or interfacing118is/are implemented as a combination of hardware and software (e.g., used to map image data from one or more image sensors104a-nto one or more processors120a-n). Thus, the multiple processor interface108and/or interfacing118may provide a mapping112function.

The multiple processor interface108and/or interfacing118provide other functionality such as segmentation114and synchronization116. The segmentation module114segments image data from one or more image sensors104a-n. Each image data segment is sent to a processor120a-nfor processing. The synchronization module116synchronizes image data rates and processor120a-nclock rates to enable the processors120a-nto process segmented images. This enables image processing to occur in parallel and/or in real time. As the processing occurs in real time, the image data or segments may not be stored in a buffer or memory until after processing. That is, the image data and/or image data segments may be continuously streamed in real time until after processing. The image data rates may be the same or different (e.g., for each image sensor104a-n). Similarly, the processor120a-nclock rates may be the same or different.

The controller110included in the multiple processor interface108may be a hardware and/or software module used to control the operations of the multiple processor interface108and/or interfacing118. In one configuration, the controller110is implemented as a hardware module for controlling mapping112, segmentation114and/or synchronization116functionality. In another configuration, the controller110is implemented as a software module, including instructions used to control mapping112, segmentation114and/or synchronization116. In yet another configuration, the controller110module is implemented as a combination of hardware and software.

The controller110controls mapping112, segmentation114and/or synchronization116functions. For example, the controller110may determine how image data from one or more image sensors104a-nis mapped112to one or more processors120a-n, how image data is segmented114and/or how image data rates and processor120a-nclock rates are synchronized116. The controller110may also determine whether segmentation114(e.g., parallel processing) would be beneficial in a particular case. This determination may be based on considerations such as current image sensor104a-nand processor120a-nusage, the resolution and/or frame rate of image capture desired, the amount of time needed to process the image data with/without parallel processing, image sensor104a-nand/or processor120a-ncapabilities and/or power consumption, etc.

The combining module122combines processed image data segments (also referred to as “processed image segments” or “processed segments”). The combining module122may include an output interface and/or an output buffer. In one configuration, the combining module122includes an output interface that combines processed image segments before sending it to the output buffer. In another configuration, the combining module122includes an output interface that sends the processed image segments directly to the output buffer. In yet another configuration, the combining module122does not include an output interface, but includes an output buffer where the processed image segments are sent from each of the processors120a-n.

For example, assume that a high resolution image is captured by image sensor A104a. Also assume that the controller110determines that image data segmentation114would be beneficial. The mapping module112maps the image data to multiple processors120a-n, which image data is segmented114or divided into multiple segments or slices and synchronized116to the clock rate of each processor120a-n. Each processor120a-nprocesses its respective segment of the image data and outputs a processed image segment to the combining module122. The combining module122combines the processed image segments to produce a processed image124. This procedure may be followed to produce multiple processed images124.

If the electronic device102(e.g. controller110) determines that it would not be beneficial to segment the image data for parallel processing, the electronic device102may not map image data to non-dedicated processors120. In that case, that is, the electronic device102may map image data from image sensors104a-nonly to dedicated processors120(e.g., perform typical operation). In summary, the processors120a-nmay be used individually (e.g., each processor120a-nbeing dedicated to a specific image sensor104a-n) or may be used in parallel to process an image and achieve a higher throughput. In general, N processors120a-nmay be used, where N≧2. Thus, the image data may be split into N strips (e.g., vertical or horizontal) or regions for parallel real-time processing by the processors120a-n. In one configuration, image sensors104send out image data in a raster scan order. Thus, the image data may be segmented vertically. For example, as the image data is received on a line, a first group of pixels may be sent to one processor and a second group of pixels may be sent to a second processor, etc.

A more specific example of parallel image processing using multiple processors120follows. In this example, assume that the electronic device102has two processors: processor A120aand processor B120b. Also assume that processor B120bis not dedicated to image sensor A104a. Image sensor A104astreams real-time image data into the electronic device102(e.g., into the multiple processor interface108and/or interfacing118). The multiple processor interface108and/or interfacing118streams a left portion (e.g., vertical strip) of the image data to processor A120awhile sending a right portion of the image data to processor B120b. In this example, the multiple processor interface108and/or interfacing118may be implemented as two crop modules, where the first crop module selects pixels (e.g., a first segment) for processor A120awhile the second crop module selects pixels (e.g., a second segment) for processor B120b. Processor A120aand processor B120bprocess the image data segments in real time and in parallel. The combining module122may send the processed image segments directly to an output buffer or combine the processed image segments before sending it to the output buffer. Alternatively, the combining module122may not include an output interface, in which case the processors120directly write the processed image segments to the output buffer.

FIG. 2is a flow diagram illustrating one configuration of a method200for parallel processing images using multiple processors120. An electronic device102obtains202image data from one or more image sensors104. For example, one or more image sensors104capture and provide image data. The electronic device102maps204the image data to two or more processors120. In one configuration, the multiple processor interface108and/or interfacing118includes several multiplexers that can be controlled to map204the image data to two or more processors120.

The electronic device102segments206the image data to produce segmented image data or segments. For example, a controller110may control a segmentation114module or function that splits or segments206the image data. The image data may be segmented206into two or more segments. Each image data segment may comprise a number of pixels, for example. The image data segments may be the same size or different sizes. In one configuration, the image data segments include overlapping data (e.g., “pads” or “padding”) in order to avoid unwanted artifacts at the “seam(s)” of the segments.

The electronic device102synchronizes208the image data to the clock rates of two or more processors120. When the image data is segmented206, it may be split into two or more image data streams. In order for processing to occur in real time, the electronic device102may synchronize208the image data streams to the clock rates of the two or more processors120. This allows processing to proceed in real time. In other words, the image data stream rates may be adjusted to match that of each processor120. In one configuration, synchronization may be accomplished using First In, First Out (FIFO) buffers. This may allow the image data to be written at one rate by the source (e.g., image sensor104data) and read out at a different rate by the sink (e.g., processors120). The FIFO buffer may be sized to be large enough such that the data rates do not cause overflows in the buffer.

The electronic device102processes210the segmented image data (also referred to as “image data segments” or “segments”) to produce processed image segments (also referred to as “processed segments”). For example, each processor120may process210an image data segment. As mentioned above, some examples of processing210include cropping, scaling, converting image formats, color interpolation, color processing, filtering images (e.g., spatially filtering images), correction for lens artifacts or defects, etc. The processing210may occur in parallel and/or in real time. The image data segments may be sent from one or more image sensors104(that captured the image data) to one or more processors120that are not dedicated to the one or more image sensors104. In other words, one or more non-dedicated processors120may be used in addition to or alternatively from one or more dedicated processors120to process the image data segments. Processing210the segmented image data produces processed image segments. The electronic device102combines212the processed image segments in order to produce a processed image124. The processed image124may be displayed, stored and/or transmitted, for example.

FIG. 3is a flow diagram illustrating a more specific configuration of a method300for parallel image processing using multiple processors120. An electronic device102obtains302image data from one or more image sensors104. For example, one or more image sensors104capture image data and provide the image data to the electronic device102.

The electronic device102may determine304whether it would be beneficial to segment the image data for parallel processing. As discussed above, this determination304may be based on considerations such as current image sensor104and processor120usage, the resolution and/or frame rate of image capture desired, the amount of time needed to process the image data with/without parallel processing, image sensor104and/or processor120capabilities and/or power consumption, etc. In one example, a controller110determines whether a dedicated processor120alone would be capable of processing the image data within a given amount of time (e.g., at a given resolution and/or frame rate). If the dedicated processor120is incapable of processing (or unable to process) the image data within the given amount of time or at a desired frame rate, the controller110determines304that it would be beneficial to segment the data for parallel processing. In other words, the number of processors120used to process the image data may be dynamically adjusted (e.g., added or removed) based on the incoming or current workload (e.g., amount, resolution and/or frame rate of image data).

In another example, the controller110bases its determination304on current processor120usage. Assume that one processor120is busy processing video images at a particular frame rate and does not have additional capacity to process a still image. In this case, the controller110may determine304that it would not be beneficial to segment the image data for parallel processing. Alternatively, the controller110may determine304that it would be beneficial to segment the image data in this case. For instance, the controller110may determine that mapping some of the video processing to another processor120and mapping the still image processing to both processors120will maintain the video stream and process the still image more quickly than using dedicated processors120. Thus, the electronic device102may dynamically add or group processors120for video or image processing when it would be beneficial.

In yet another example, the controller110determines304that segmenting the image data for parallel processing of an image will be beneficial by conserving power resources (e.g., battery). In this example, assume that a dedicated processor120is capable of processing the image in an acceptable amount of time, but doing so will require the dedicated processor120to run at a high clock rate, thus dissipating a relatively large amount of power. The controller110may determine304that segmenting the image data for parallel processing would be beneficial in this case by running two processors120at lower clock rates that use less power overall than running the dedicated processor at the high clock rate.

If the electronic device102determines304that it would not be beneficial to segment the image data for parallel processing, the electronic device102may process306image data from each image sensor104using its dedicated processor(s)120. For example, the controller110may map image data from each image sensor104to its dedicated processor(s)120and process306the image data accordingly.

If the electronic device102determines304that it would be beneficial to segment the image data for parallel processing, the electronic device102may determine308a mapping. That is, the controller110may determine308which processor or processors120to map the image data to. In one configuration, the controller110determines which (and how many) processors120are needed to process the image data at the current resolution and/or frame rate. The mapping determination308may be based on considerations such as current image sensor104and processor120usage, the resolution and/or frame rate of image capture desired, the amount of time needed to process the image data with/without parallel processing, image sensor104and/or processor120capabilities and/or power consumption, etc.

For example, if a first processor120alone or individually is currently incapable of processing the image data at the desired resolution and/or frame rate, the controller110may add or map the image data to additional processors120until enough processing power is available to process the image data. For instance, the number of processors120used to process the image data may be dynamically adjusted (e.g., added or removed) based on the incoming or current workload (e.g., amount, resolution and/or frame rate of image data). Other approaches to determine308a mapping may be used. For example, the controller110may map the image data to as many processors120as possible or to a combination of processors120that minimizes power consumption or average processing rate.

In another configuration, the mapping determination308may be based on current processor120usage or workload. Assume that one processor120is busy processing video images at a particular frame rate and does not have additional capacity to process a still image. In this case, a mapping may be determined308that maps the still image processing to one or more other processors120. Alternatively, the mapping may be determined308such that some of the video processing is mapped to another processor120and the still image processing is mapped to both processors120in order to maintain the video stream and process the still image. Thus, the electronic device102may map308processors120for video or image processing.

In yet another example, the mapping is determined308based on conserving power resources (e.g., battery). In this example, assume that a dedicated processor120is capable of processing the image in an acceptable amount of time, but doing so will require the dedicated processor120to run at a high clock rate, thus dissipating a relatively large amount of power. The mapping may be determined308such that the image data is mapped to multiple processors120running at lower clock rates that use less power overall than running the dedicated processor at the high clock rate.

The electronic device102maps310the image data to two or more processors120. For example, the electronic device102or controller110uses an array of multiplexers, switches and/or other addressing scheme to map or route the image data to two or more processors120.

The electronic device102may determine312a segmentation. For example, the electronic device102or controller110may base the segmentation on processor120capacity, speed (or processing rate), current usage, etc. In one simple example, the controller110determines312a proportionate segmentation based on processor120capacity or capability. For instance, assume that one processor120has twice the capacity or capability of another processor120. The controller110may segment the image data in a 2:1 ratio, providing twice the number of pixels to one processor120compared to the other processor120. In another example, the controller110segments the image data such that up to the maximum processing capacity or capability of each successive processor120is used until sufficient processing capacity or capability is allocated to process the image data. It should be noted that the image data may be segmented into vertical strips, horizontal strips and/or other regions that are a subset of the image data. In one configuration, image sensors104send out image data in a raster scan order. Thus, the image data may be segmented vertically. For example, as the image data is received on a line, a first group of pixels may be sent to one processor and a second group of pixels may be sent to a second processor, etc.

The segmentation determination312may include overlapping image data or “padding.” More specifically, if the electronic device102(e.g., processors120) uses filtering (e.g., spatial filtering), then the image data segments sent to the parallel processors120will need to take care of seam or edge conditions. This means that overlapping image data segments (e.g., with “padding”) may be sent to the parallel processors120. The amount of overlap may be determined by the amount needed to support spatial or filtering structures in the processing pipeline. For example, assume that a processor120uses filtering (e.g., spatial filtering) in the horizontal direction with vertical segmentation. If the electronic device102uses a 3×3 filtering kernel, then along each segmentation seam, the left and right segments each need an overlap of one pixel. However, if each of the processors120processes the image data on a per-pixel basis, then no overlap may be needed.

The electronic device102segments314the image data to produce segmented image data (e.g., two or more image data segments). For example, the electronic device102or a segmentation module114segments314the image data according to the segmentation determined312by the controller110. The electronic device102may also segment314the image data with overlapping segments or padding to avoid edge artifacts at the segment seam(s). The electronic device102or segmentation module114may segment the image data by discarding or cropping image data that does not correspond to a particular segment or by accepting or receiving image data that corresponds only to a particular segment (with applicable padding or overlap). The electronic device102or segmentation module114may segment the image data in vertical strips, horizontal strips or other regions, for example.

If the processors120utilize per-pixel processing or all processing is done on a per-pixel basis, the image segments sent to the parallel processors120may be non-overlapping. In this case, no overlap may be needed because no seams or edge artifacts may result from the processing. However, if the processors120utilize filtering (e.g., spatial filtering) in the horizontal direction (e.g., with vertical segments) or vertical direction (e.g., with horizontal segments), then the image segments sent to the parallel processors120may need an overlap in image data to properly handle seam or edge conditions (e.g., to avoid artifacts).

The electronic device102synchronizes316the image data to clock rates of the two or more processors120. For example, the electronic device102or a synchronization module116adjusts the rate of the image data stream for an image data segment to match the clock rate of the processor120that the image data segment has been mapped310to. In one configuration, this may be accomplished using FIFO buffers as described above.

The electronic device102processes318the segmented image data using the two or more processors120to produce processed image segments (e.g., “processed segments”). As discussed above, processing318the segmented image data may include cropping, scaling, converting image formats, color interpolation, color processing, filtering images (e.g., spatially filtering images), correction for lens artifacts or defects, etc. Processing318the segmented image data yields processed image segments.

The electronic device102or combining module122combines320the processed image segments using an output interface and/or buffer. Combining320the processed image segments produces a processed image124. For example, the electronic device102may include an output interface that combines320the processed image segments before sending the processed image124to an output buffer. In another configuration, an output interface may not be used or needed. In this case, the processors120may directly write the processed image segments to the output buffer. Each pipeline (e.g., processor120) may crop out extra pixels from having overlapped portions or “padding,” if any. Alternatively, the combining module122(e.g., output interface) may remove extra pixels as it combines320the processed image segments. In another configuration, the combining module122combines320the overlapping pixels with pixels in an adjacent processed segment.

The electronic device102may output322the processed image124for display, storage and/or transmission. For example, the electronic device102may store the processed image124in memory. Alternatively or in addition, the electronic device102may display the processed image124and/or transmit the processed image124to another device (e.g., another electronic device, wireless communication device, computer, etc.).

FIG. 4is a block diagram illustrating one example of parallel image processing using multiple processors. In this example, image sensor A404aobtains or captures an image426. Image sensor A404athen provides image data428to a multiple processor interface408. The multiple processor interface408maps the image data428to processor A420aand processor B420band segments the image data428into segment A430aand segment B430b. The multiple processor interface408also synchronizes segment A430aand segment B430bwith respective processor A420aand B420bclock rates. The multiple processor interface408provides segment A430ato processor A420aand segment B430bto processor B420b. Processor A420aprocesses segment A430ato produce processed segment A432aand processor B420bprocesses segment B430bto produce processed segment B432b. Processed segment A432aand processed segment B432bare provided to the output interface422. The output interface422combines the processed segments A432aand B432bto produce the processed image424.

A more specific example follows. Assume that a 360 megapixels per second (MP/sec) throughput is required to process a 4000×3000 resolution image sensor input at 30 FPS. Assume that the only filtering operation in the pipeline is a 3×3 Finite Impulse Response (FIR) spatial filter. No sensor blanking is assumed in this example. Further assume that processor A420ais an image signal processor (ISP) with a performance of 260 MP/sec up to 3504 pixel line width. This type of image signal processor (ISP) may be typically used for an 8 MP camera.

The performance that processor B420bwould need to provide in order to process the image data428at the current resolution and frame rate may be determined. While processor A420acan process up to 3504 pixels in a line, it is limited by its processing throughput since 3504*3000*30 FPS is greater than 260 MP/sec. Processor A420acan only process 4000 pixels*(260 MP/sec)/(360 MP/sec)≈2888 pixel width. Since there is also a 3×3 spatial filter, one extra column of padding is also needed. Thus, segment A430a(the input to processor A420a) may be 2888 pixels wide by 3000 pixels high, while processed segment A432a(the output from processor A420a) is 2887 by 3000 pixels. Thus, processor B420bmust be able to process 1114 pixels per line=4000−2887+1 padding column. Thus, processor B's420bthroughput must be at least 1114 pixels*3000 pixels*30 FPS≈101 MP/sec. If processor B420bdoes not have this performance, one or more processors420may be added.

By combining two or more processors, image data from larger resolution sensors can be processed than could be processed by the individual processors. This benefit of the systems and methods disclosed herein is illustrated in Table (1) below. For example, assume that processor A420ais able to process 5.04 MP images and processor B420bis able to process 3.15 MP images. According to Table (1), processor A420aand processor B420btogether could process image data from a 15.27 MP resolution sensor. This is because processor A420aprocesses one part of the image data (e.g., a left segment) and processor B420bprocesses another part (e.g., a right segment) of the image data. Thus, multiple processors may process image data from high resolution image sensors. In Table (1), “Width” (in pixels) is abbreviated as “W” and “Height” (in pixels) is abbreviated as “H” for convenience.

TABLE 1Processor AProcessor BMPWHMPWHMPWHMPWHMPWH1.92160012003.15204815363.87227217045.04259219441.92160012007.08307223049.293520264010.513744280812.39406430483.15204815369.293520264011.813968297613.184192314415.27451233843.872272170410.513744280813.184192314414.634416331216.82473635525.042592194412.394064304815.274512338416.824736355219.17505637925.952816211213.794288321616.824736355218.454960372020.91528039607.993264244816.824736355220.165184388821.935408405624.61572842969.983648273619.665120384023.255568417625.165792434428.026112458412.004000300022.465472410426.285920444028.316144460831.346464484813.334216316224.275688426628.246136460230.346360477033.476680501015.934608345627.726080456031.966528489634.196752506437.517072530416.824736355228.906208465633.236656499235.506880516038.887200540021.685376403235.176848513639.927296547242.417520564046.1078405880
It should be noted that image signal processors (e.g., processors420) may be limited by the width of the image they can process. For example, processor A420awith a width of 1600 and processor B420bwith a width of 2048 may process an image that has a width of almost 1600+2048, which is approximately 9 MP. In Table (1), the combined widths are not illustrated as a simple sum of widths to account for some padding, which has been set to 128 in this example. More specifically, the combination of processor A420awidth of 1600 and processor B420bwidth of 2048 would sum to 3648, but is illustrated with a width of 3520 to account for padding. It should also be noted that the heights illustrated are determined assuming a picture aspect ratio of 4:3, which may be typical for image sensors.

FIG. 5is a block diagram illustrating another example of parallel image processing using multiple processors. More specifically,FIG. 5illustrates a single image sensor504/multiple image signal processor520case. An image sensor504captures or obtains image data528. In this example, the image sensor504provides image data528as a single high-bandwidth data stream534to a mapping, segmentation and synchronization module536. The mapping, segmentation and synchronization module536maps the image data528to multiple processors520a-n, segments the image data528into segment A530a, segment B530b, segment C530cand so on, up to segment N530n. That is, the image data528is segmented into two or more segments530. Each segment530is provided to parallel processing image signal processors540as a low bandwidth data stream538a-n. Thus, multiple low bandwidth data streams538a-nare sent to parallel processing image signal processors540. More specifically, segment A530ais provided to image signal processor A520aas low bandwidth data stream A538a, segment B530bis provided to image signal processor B520bas low bandwidth data stream B538b, segment C530cis provided to image signal processor C520cas low bandwidth data stream C538cand so on up to segment N530nbeing provided to image signal processor N520nas low bandwidth data stream N538n. The data mapping, segmentation and synchronization module536also synchronizes each low bandwidth data stream538a-nto the respective clock rate of each image signal processor520a-n. As illustrated inFIG. 5, the systems and methods disclosed herein provide data rate reduction through parallel processing using multiple processors.

Each of the image signal processors520a-nprovides a processed segment532a-nto a combining module522. The combining module522combines the processed image segments A-N532a-ninto a processed image524. As illustrated by the example inFIG. 5, image processing may be accomplished in parallel and in real time according to the systems and methods disclosed herein.

FIG. 6is a block diagram illustrating more detail for one configuration of a multiple processor interface608. One or more image sensors604a-mmay be coupled to a multiple processor interface608. The multiple processor interface608may be coupled to two or more processors620a-n. The multiple processor interface608may be implemented as a hardware and/or software module. For example, the multiple processor interface608may be implemented as an Integrated Circuit (IC) including components used to implement the systems and methods disclosed herein. Alternatively, the multiple processor interface608may be implemented as a software module including instructions or code used to implement the systems and methods disclosed herein. Alternatively, the multiple processor interface608may be implemented as a combination of hardware and software. Thus, the multiple processor interface608is described in terms of functionality.

The multiple processor interface608may include a multiplexer array644, two or more interface modules650a-nand two or more synchronizers654a-n. The multiplexer array644may provide mapping612functionality. For example, the multiplexer array644may include one or more data multiplexers (labeled “Data Mux” inFIG. 6for convenience)646a-mand one or more clock multiplexers648a-m(labeled “Clock Mux” inFIG. 6for convenience). The data multiplexers646a-mmap image data628a-mto the two or more processors620a-n. More specifically, the data multiplexers646a-mmay map image data628a-mfrom any of the one or more image sensors604a-mto any of the two or more processors620a-n.

The clock multiplexers648a-mare used to map sensor clock signals642a-mto the interface modules650a-nand to the synchronizer modules654a-n. The sensor clock signals642a-mmay indicate a frame rate or rate at which image data628a-mis being captured. In other words, the clock multiplexers648a-mmay map the sensor clock signals642a-mto the interface blocks650a-nand the synchronizers654a-nas clock signals652a-n. For example, sensor clock B642bmay be mapped to interface A650aand synchronizer A654aas clock signal A652a.

The interface modules650a-nprovide segmentation614a-nfunctionality. For example, the interfaces650a-nmay be implemented as “croppers” that segment the image data628a-mfrom the image sensors604a-m. The interface modules650a-nmay use the clock signals652a-n. For example, the interface modules650a-nmay be implemented in hardware, such that the clock signals652a-nare used to run the circuits. The sensor clocks642a-mmay be synchronized (respectively, for example) with the image data628a-m. Thus, the clock signals652a-nmay allow the hardware to be timed such that each cycle corresponds to each data element from the image sensors604. In this way, the interface modules650a-nmay “understand” when data is delivered and the timing of the data. The timing of the data may be used to synchronously drive logic that used for operations like cropping. Generally, the clock signals652a-nmay provide one or more timing signals for segmentation logic to understand when valid data is delivered to it from the image sensors604a-m.

The interface modules650a-nprovide the segmented image data to the synchronizers654a-n. The synchronizers654a-nprovide synchronization616a-nfunctionality. The synchronizers654a-nmay use the clock signals652a-nand the processor clock signals656a-nin order to synchronize the segmented image data to the clock domain of the processors620a-n. In this way, the segmented image data may be processed in parallel and in real time by the processors620a-n. In one configuration, each processor620a-nruns at a different clock rate. Thus, synchronizing616a-nthe segmented image data into the clock domain of each processor620a-nmay be needed to coordinate the proper processing of each image segment. As illustrated inFIG. 6, each processor620a-noutputs processed image data658a-n. It should be noted that according to the systems and methods herein, there may be a different number of image sensors604a-mthan processors620a-n. Alternatively, there may be the same number of image sensors604a-mas processors620a-n.

FIG. 7is a diagram illustrating one example of image data segmentation. Specifically, vertical segmentation of image data for processing using two processors120is illustrated. Active pixels774and blanking764are two regions of image data provided by an image sensor104. Active pixels774are pixels in the image data used to produce an image. In other words, the active pixels774are pixels to be processed in order to produce a processed image124. Blanking764is a region surrounding the active pixels774. The blanking region764may comprise a region that is not part of the active pixel region774. For example, the image sensor104may or may not send blanking region764pixels. For instance, the blanking region764may be used in between frames and/or between the lines in within a frame to provide image sensor104circuitry with a number of cycles to finish its work and prepare for handling the next segment of data. During this time, there may not be any valid data to be presented, thus resulting in a blanking region764.

InFIG. 7, several letters represent various dimensions of image data. More specifically, m760represents the horizontal dimension of a first segment including blanking764, n762represents the horizontal dimension of a second segment including blanking764, p776represents the vertical dimension of the active pixels774and blanking764, q766represents the horizontal dimension of a first segment of active pixels774, r772represents the horizontal dimension of a second segment of active pixels and s778represents the vertical dimension of active pixels774. In this example, image data from one image sensor104is segmented into two segments for processing on two processors120. According to the example, assume that the image sensor104has a resolution (e.g., in pixels) as illustrated in Equation (1).
Sensor_Resolution=(q+r)*s(1)
An image data input rate may be defined as illustrated in Equation (2).
Input_Rate=(m+n)*p*fps  (2)
In Equation (2), fps is a frame rate (e.g., in FPS). The Input_Rate is the rate at which image data is input into a multiple processor interface108from the image sensor104.

In this example, the image data is segmented into two segments by a seam780. The first segment (of active pixels774) has dimensions of q766by s778. However, a first pad with dimensions pad1768by s778may also be processed by the first processor120. Thus, the input resolution of the first processor120is illustrated in Equation (3).
Input_Resolution1=(q+pad1)*s(3)
Accordingly, the first processor120has a processing rate as illustrated in Equation (4).
Processing_Rate1≧(m+pad1)*p*fps  (4)
And, the output resolution of the first processor120is illustrated in Equation (5).
Output_Resolution1=q*s(5)

Similarly, the input resolution, processing rate and output resolution of the second processor120are given in Equations (6), (7) and (8), respectively.
Input_Resolution2=(r+pad2)*s(6)
In Equation (6), Input_Resolution2is the input resolution of the second processor120and pad2770is the horizontal dimension of a second pad. It should be noted that the sizes of pad1and pad2may be determined by the amount of padding needed to avoid artifacts at the seam780as discussed above.
Processing_Rate2≧(n+pad2)*p*fps  (7)
In Equation (7), Processing_Rate2is the processing rate of the second processor120. Equation (8) illustrates the output resolution of the second processor120(Output_Resolution2).
Output_Resolution2=r*s(8)

FIG. 8illustrates various components that may be utilized in an electronic device802. The illustrated components may be located within the same physical structure or in separate housings or structures. The electronic device102discussed in relation toFIG. 1may be configured similarly to the electronic device802. The electronic device802includes one or more image sensors804. The one or more image sensors804may be one or more devices that capture or convert an optical signal (e.g., image) into an electronic signal or image data. Examples of image sensors804include Charge-Coupled Device (CCD) and Complementary Metal Oxide Semiconductor (CMOS) sensors.

The electronic device802includes two or more image signal processors820. The two or more image signal processors820may be directly and/or indirectly coupled to the one or more image sensors804. The two or more image signal processors820may be general purpose single- or multi-chip microprocessors (e.g., ARMs), special purpose microprocessors (e.g., digital signal processors (DSPs)), microcontrollers, programmable gate arrays, etc. The two or more image signal processors820may be used to perform image processing functions. For example, the two or more image signal processors may perform cropping, scaling (e.g., to a different resolution), image format conversion, color interpolation, color processing, image filtering (e.g., spatial image filtering), lens artifact or defect correction, etc.

The electronic device802includes a processor888. The processor888may be a general purpose single- or multi-chip microprocessor (e.g., an ARM), a special purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. The processor888may be referred to as a central processing unit (CPU). Although just a single processor888is shown in the electronic device802ofFIG. 8, in an alternative configuration, a combination of processors (e.g., an ARM and DSP) could be used.

The electronic device802also includes memory882in electronic communication with the processor888. That is, the processor888can read information from and/or write information to the memory882. The memory882may be any electronic component capable of storing electronic information. The memory882may be random access memory (RAM), read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory included with the processor, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), registers, and so forth, including combinations thereof.

Data886aand instructions884amay be stored in the memory882. The instructions884amay include one or more programs, routines, sub-routines, functions, procedures, etc. The instructions884amay include a single computer-readable statement or many computer-readable statements. The instructions884amay be executable by the processor888to implement the methods that were described above. Executing the instructions884amay involve the use of the data886athat is stored in the memory882.FIG. 8shows some instructions884band data886bbeing loaded into the processor888.

The electronic device802may also include one or more communication interfaces890for communicating with other electronic devices. The communication interfaces890may be based on wired communication technology, wireless communication technology, or both. Examples of different types of communication interfaces890include a serial port, a parallel port, a Universal Serial Bus (USB), an Ethernet adapter, an IEEE 1394 bus interface, a small computer system interface (SCSI) bus interface, an infrared (IR) communication port, a Bluetooth wireless communication adapter and so forth.

The electronic device802may also include one or more input devices892and one or more output devices894. Examples of different kinds of input devices892include a keyboard, mouse, microphone, remote control device, button, joystick, trackball, touchpad, lightpen, etc. Examples of different kinds of output devices894include a speaker, printer, etc. One specific type of output device that may be typically included in an electronic device802is a display device896. Display devices896used with configurations disclosed herein may utilize any suitable image projection technology, such as a cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), gas plasma, electroluminescence, or the like. A display controller (not shown) may also be provided for converting data stored in the memory882into text, graphics and/or moving images (as appropriate) shown on the display device896.

The electronic device802may also include a power source and/or interface898. The power source and/or interface898may provide electrical power to the electronic device802. For example, the power source/interface898may be a battery. Alternatively, the power source/interface may be a port through which electrical power may be provided. For example, the power source/interface898may be a port that accepts Alternating Current (AC) or Direct Current (DC) power. In one configuration, the power source/interface898is used to accept a power adapter that plugs into a power outlet. Alternatively, the power source/interface898may accept electrical power via a USB port. In yet another configuration, the power source/interface898wirelessly receives electrical power (e.g., using an inductive or proximity charging device).

The various components of the electronic device802may be coupled together by one or more buses, which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. For simplicity, the various buses are illustrated inFIG. 8as a bus system811. It should be noted thatFIG. 8illustrates only one possible configuration of an electronic device802. Various other architectures and components may be utilized.

FIG. 9illustrates various components that may be utilized in a wireless communication device901. The wireless communication device901may be a kind of electronic device102,802used to wirelessly communicate with other electronic devices. Examples of wireless communication devices901include cellular telephones, smart phones, Personal Digital Assistants (PDAs), e-readers, gaming systems, music players, netbooks, wireless modems, laptop computers, tablet devices, etc. The electronic device102discussed in relation toFIG. 1may be configured similarly to the wireless communication device901.

Many of the components included in the wireless communication device901may be configured and function similarly to the components described in relation toFIG. 8. For example, the wireless communication device may include one or more image sensors904, two or more image signal processors920, memory982including instructions984aand/or data986a, a processor988loading instructions984band/or data986bfrom memory982, one or more communication interfaces990, one or more input devices992, one or more output devices994, such as a display device996and a power source/interface998. The wireless communication device901may additionally include a transmitter905and a receiver907. The transmitter905and receiver907may be jointly referred to as a transceiver903. The transceiver903may be coupled to one or more antennas909for transmitting and/or receiving wireless signals. Similar to the electronic device802described above in relation toFIG. 8, the wireless communication device901may include one or more buses illustrated as a bus system911inFIG. 9. The bus system911may couple the described components together, allowing coordinated operation.

As used herein, the term “wireless communication device” generally denotes an electronic device (e.g., access terminal, client device, client station (STA) etc.) that may wirelessly connect to another electronic device (e.g., base station). A wireless communication device may alternatively be referred to as a mobile device, a mobile station, a subscriber station, a user equipment (UE), a remote station, an access terminal, a mobile terminal, a terminal, a user terminal, a subscriber unit, etc. Examples of wireless communication devices include laptop or desktop computers, cellular phones, smart phones, wireless modems, e-readers, tablet devices, gaming systems, etc. Wireless communication devices may operate in accordance with one or more industry standards such as the 3rdGeneration Partnership Project (3GPP). Thus, the general term “wireless communication device” may include wireless communication devices described with varying nomenclatures according to industry standards (e.g., access terminal, user equipment (UE), remote terminal, etc.).

The functions described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term “computer-readable medium” refers to any available medium that can be accessed by a computer or processor. By way of example, and not limitation, such a medium may comprise RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. It should be noted that a computer-readable medium may be tangible and non-transitory. The term “computer-program product” refers to a computing device or processor in combination with code or instructions (e.g., a “program”) that may be executed, processed or computed by the computing device or processor. As used herein, the term “code” may refer to software, instructions, code or data that is/are executable by a computing device or processor.