Foveated image capture for power efficient video see-through

Generating an image stream may include obtaining image data from a camera, identifying a first subset of the image data including a region of interest, identifying a second subset of the image data different than the first subset of the image data, processing the first subset of image data by a first processing pipeline to obtain a first processed set of image data, processing the second subset of image data by a second processing pipeline to obtain a second processed set of image data, wherein the second processing pipeline processes at a lower quality than the first processing pipeline, and combining the first processed set of image data and the second processed set of image data to obtain a processed image frame.

BACKGROUND

This disclosure relates generally to the field of digital image processing, and more particularly to the field of foveated image capture for power efficient video see-through. There are many uses for processing a video feed, including live streaming and use in augmented reality (AR) contexts. Generally, one of the goals in processing image data for a video feed is to reduce power cost of a video camera. Processing all image data at a single quality level can use a high level of resources so as to affect system performance. Additionally, processing all image data at a high quality level can lead to latency issues and could lead to an unpleasant user experience. Further, processing image data at a lower quality level often can also be detrimental. Image quality at a lower resolution may also lead to an unpleasant user experience.

SUMMARY

In one embodiment, a method for generating an image stream is described. The method includes obtaining image data from a camera, identifying a first subset of the image data comprising a region of interest, identifying a second subset of the image data different than the first subset of the image data, processing the first subset of image data by a first processing pipeline to obtain a first processed set of image data, processing the second subset of image data by a second processing pipeline to obtain a second processed set of image data, wherein the second processing pipeline processes at a lower quality than the first processing pipeline, and combining the first processed set of image data and the second processed set of image data to obtain a processed image frame. In another embodiment, the method may be embodied in computer executable program code and stored in a non-transitory storage device. In yet another embodiment, the method may be implemented in an electronic device having image capture capabilities.

DETAILED DESCRIPTION

This disclosure pertains to systems, methods, and computer readable media for foveated image capture for power efficient video. By applying foveated image processing techniques to video capture, image data may be captured and processed in a more efficient manner. In general, techniques are disclosed for obtaining video image data of the environment and processing the image data in a way that a particular region of the captured scene is processed at a higher quality than another portion of the scene, such as a background. Thus, a particular frame of the video image data may have one portion, such as a portion of the image containing a region of interest, at a higher quality than other portions of the image, which may not be as interesting to a viewer. In order to capture a unique picture, but selectively process a region that is deemed of higher importance to a user at a higher quality than a region considered of lesser importance, two data flows from a video image data from a single camera may be utilized.

In one or more embodiments, generating a frame of an image stream may include obtaining image data from a camera, such as image data for a frame of a video stream. A region of interest in the scene may be identified in any number of ways, and the portion of the image data that includes the region of interest may be processed using a first processing pipeline. A second portion of the image data, which may include, for example, the portion of the image data not including first portion, or the entire image data for the frame (capturing the entire scene), may be processed by a second processing pipeline, which may produce lower quality image data, but may be more efficient than the first processing pipeline. The two sets of processed image data may then be combined to form a composite image data, or a processed image frame. According to one or more embodiment, by only processing a portion of the image data that includes the region of interest for each frame, the resulting video image data may be processed in an efficient manner.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of this disclosure's drawings represent structures and devices in block diagram form in order to avoid obscuring the novel aspects of the disclosed embodiments. In this context, it should be understood that references to numbered drawing elements without associated identifiers (e.g.,100) refer to all instances of the drawing element with identifiers (e.g.,100A and100B). Further, as part of this description, some of this disclosure's drawings may be provided in the form of a flow diagram. The boxes in any particular flow diagram may be presented in a particular order. However, it should be understood that the particular flow of any flow diagram or flow chart is used only to exemplify one embodiment. In other embodiments, any of the various components depicted in the flow diagram may be deleted, or the components may be performed in a different order, or even concurrently. In addition, other embodiments may include additional steps not depicted as part of the flow diagram. The language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the disclosed subject matter. Reference in this disclosure to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment, and multiple references to “one embodiment” or to “an embodiment” should not be understood as necessarily all referring to the same embodiment or to different embodiments.

It should be appreciated that in the development of any actual implementation (as in any development project), numerous decisions must be made to achieve the developers' specific goals (e.g., compliance with system and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such development efforts might be complex and time consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art of image capture having the benefit of this disclosure.

Referring toFIG. 1, a simplified block diagram of an electronic device100is depicted in accordance with one or more embodiments of the disclosure. Electronic device100may be part of a multifunctional device such as a mobile phone, tablet computer, personal digital assistant, portable music/video player, or any other electronic device that includes a camera system and a display. Further, electronic device100may be part of a larger system of components that includes a camera and a display. Electronic Device100may be connected to other devices across a network such as other mobile devices, tablet devices, desktop devices, as well as network storage devices such as servers and the like. Electronic device100may be configured to capture image data corresponding to a scene and use the captured image data to render views on a display viewable by a user.

Electronic Device100may include a processor130. Processor130may be a central processing unit (CPU). Processor130may alternatively, or additionally, include a system-on-chip such as those found in mobile devices and include zero or more dedicated graphics processing units (GPUs). Also connected on the bus may be image processing module155, which may be a software or hardware module which may be used for foveated image capture. Electronic Device100may also include memory140and storage150. Memory140and storage150may each include one or more different types of memory, which may be used for performing device functions in conjunction with processor130. For example, memory140may include cache, ROM, and/or RAM. Memory140may store various programming modules during execution. In one or more embodiments, storage150may comprise cache, ROM, RAM, and/or non-volatile memory, and may store video image data185. Video image data185may include raw image data captured by camera110and/or processed image data, such as a foveated image data generated by the image processing module155processing the raw image data.

Electronic device100may also include one or more cameras110. Each camera110may include an image sensor, a lens stack, and other components that may be used to capture images. Further, in some embodiments image sensors may be shared between different lens stacks. In one or more embodiments, cameras110may be configured to capture still pictures, or video feed.

Electronic device100may include one or more sensors175, which may provide information about a surrounding environment. For example, sensors175may include sensors configured to detect brightness, depth, location, and other information regarding the environment. Electronic device100may also include a display180, which may be an additive display. For example, display180may be a transparent or semi-opaque display, such as a heads-up display, by which an image may be projected over a transparent surface. Thus, display180may be comprised of a projector and a surface, or may just include the projector. Further, display180may be a transparent display, such as an LCD display and/or a head mounted display. Electronic device100may additionally include I/O devices120, such as speakers and the like. In one or more embodiments, the various I/O devices120may be used to assist in image processing to generate a foveated image feed, such as by receiving user input to identify a region of interest in a scene. According to one or more embodiments, I/O devices120may additionally include a touch screen, mouse, track pad, and the like.

Memory140may include instructions, such as computer readable code executable by processor130to perform various actions. For example, image processing module155may include instructions that cause electronic device100to assist in processing image data captured by camera110to generate a foveated image feed. Image processing module155may determine a region of interest and cause a portion of the image data received from the camera110that include the region of interest to be processed at a higher quality than image data that does not include the region of interest. In one or more embodiments, the image processing module155may determine a region of interest based on data received from sensors175, which may indicate, for example, a portion of a scene captured by camera110that is more dynamic, and thus perhaps more interesting, than another portion of the image data. In an alternative embodiment, the image processing module155may request and/or receive user input indicating a portion of a scene which includes a region of interest. Further, image processing module155may generate compose images based on the processed image data from the first and second pipelines to generate the foveated image.

FIG. 2shows, in flowchart form, a method for generating a foveated image frame, according to one or more embodiments. Although the various actions are depicted in a particular order, in some embodiments the various actions may be performed in a different order. In still other embodiments, two or more of the actions may occur simultaneously. According to yet other embodiments, some of the actions may not be required or other actions may be included. For purposes of clarity, the flowchart will be described with respect to the various components ofFIG. 1. However, it should be understood that the various actions may be taken by alternative components, according to one or more embodiments.

The flowchart begins at205, where the image processing module obtains image data of a real environment. In one or more embodiments, the image data may be raw image data obtained using camera110, and may be part of a video feed. Thus, according to one or more embodiments the camera110will read out image data for a full field of view. The video feed may be captured in real time such that the foveated image is rendered as the video feed is captured. Alternatively, the video feed may be previously captured, and may be obtained from storage, such as video image data185in storage150, or a network storage accessible by electronic device100across a network. The flowchart continues at210, and the image processing module155determines image data corresponding to a particular frame.

The flowchart continues at215and the image processing module155determines a region of interest in the frame. In one or more embodiments, the region of interest may be determined in the sensor, for example by control from the image processing module155or the processor130. Further, in one or more embodiments, the region of interest may be determined in the processor130. In addition, the sensor could also incorporate different readout mechanisms for different pixels within each ROI such that one ROI readout is configured for optimal imaging performance and the other ROI(s) are configured for readout with a more power-frugal lower performance.

According to one or more embodiments, the region of interest may correspond to a portion of the captured scene in which action is taking place, or which otherwise includes content that would be a point of focus in the scene. The region of interest may be determined in any number of ways. For example, the image processing module155may receive sensor data from sensors175that indicate where movement in the scene is occurring. In addition, sensors175may capture depth information, for example, that identifies a foreground object in the scene. Image processing module155may determine that a portion of the image data corresponding to the region in which the action is captured contains the region of interest. A region of interest may correspond to a portion of the image data that captures an image of a region associated with relatively high perception value in a scene.

Further, in one or more embodiments, the region of interest may vary in size based on a number of factors. For example, the size of the region of interest may be proportional to available resources. That is, if there is sufficient available power or compute cycles, the region of interest may be larger. Conversely, if power resources, available compute cycles are low, then a smaller region of interest may be utilized. As another example, the size of the regions of interest may be related to a confidence value for the region of interest. As an example, a confidence value may be determined corresponding to the likelihood that the determined region of interest should actually be the region of interest (e.g., the determined region of interest is actually associated with relatively high perception value in a scene). As such, if a determined region of interest is associated with a high confidence value, then a smaller region of interest may be utilized because the smaller region is more likely to have a relatively high perception value.

In one or more embodiments, the sensor data from sensors175may also be used to determine a view direction of a user. For example, the sensors175may include an eye tracking camera or other type of sensor utilized to track a gaze direction of a user, or otherwise provide data such that a gaze direction of a user may be determined. Thus, the image processing module155may determine a location in a scene in which a user is gazing, and the image processing module155may identify a portion of the image data corresponding to image data for the region of the scene at which the user is gazing. The identified portion of the image data may be determined to be the region of interest. In another example, the image processing module155may determine a region of interest to be a default region, such as a central portion of image data for a captured scene. As another example, the region of interest may default based on a predetermined optimal foveation based on particular characteristics of the electronic device100, or another device utilized for image capture or display. Further, in another embodiment, the region of interest may be determined based on user input. As an example, image processing module155may utilize user input received from I/O devices120to determine a region of interest. For example, a user may select a particular portion of the captured scene as the region of interest by selecting the region on a display such that future image data for that region is considered the region of interest.

In one or more embodiments, the region of interest may also be determined based on a context in which the image data is to be used. As an example, if the electronic device100is utilized for augmented reality (AR) purposes or other computer-generated reality (CGR) purposes, a portion of the image over which a virtual object is to be rendered may be determined to be a region of interest. Said another way, the additional virtual data may be generated and displayed over the image of the real environment, and the area over which the virtual data is to be displayed may be determined to be an area at which a user is likely to focus, and thus be a region of interest, according to one or more embodiments.

The flowchart continues at220, and the image processing module155processes the image data based on the location of the region of interest to obtain a composite image frame. As will be described in further detail below with respect toFIG. 3, the image processing module155may direct that a first subset of the image data that includes the region of interest be processed through a first processing pipeline, whereas a second subset of the image data is processed through a second processing pipeline which is more efficient than the first processing pipeline. Said another way, the image processing module155may create two data flows, including a first data flow capturing the region of interest to be rendered at a higher resolution, and a second data flow capturing additional image data of the scene which is to be rendered at a lower resolution. The terms “low,” “lower,” “high,” “higher,” and the like may be used herein to reference resolution or other quality measures of captured image data, and do not refer to any particular quality or resolution measure, but are used to contrast the different resolutions or quality levels that may be rendered by the electronic device100. In one or more embodiments, processing the second subset of image data in the second processing pipeline may include the image processing module155downsampling the second subset of image data and directing the downsampled second subset of image data to be processed in the first processing pipeline. Thus, the second subset of data may be processed in a more efficient manner because only a portion of the second subset of data (i.e., the downsampled data) may be processed, rather than the entirety of the second subset of data. In general, power consumption is proportional to the total pixels per frame and the frame rate. Because resolution increases as PPD increases, in one or more embodiments, PPD may be maximized across image data for a frame for a region of interest.

The flowchart continues at225and the composite frame is generated. According to one or more embodiments, generating the composite frame may include combining the processed first subset of image data and the processed second subset of image data to generate the processed image frame. The processed image frame is foveated because the region of interest will be processed at a higher quality level than other regions of the image.

The flowchart continues at230and a determination is made regarding whether the image processing module155identifies image data for additional frames. If so, then the flowchart continues at210and the image data is identified for the next frame and the process repeats. In one or more embodiments, the region of interest may be the same as when the image data prior frame was processed. Thus, when215is repeated, the region of interest may be determined to be the same as the prior frame, or may be different based on new information, such as sensor data or user input. In one or more embodiments, the determined region of interest for a given frame may default to the determined region of interest for a previous frame.

According to one or more embodiments, the process continues until at230a determination is made that there are no additional frames. Then the flowchart ends. In one or more embodiments, the generated composite frames may be displayed, for example on display180. The generated composite frames may be displayed as they are generated, or upon the determination that there are no additional frames at230.

FIG. 3shows, in flowchart form, a method for generating a composite image frame, according to one or more embodiments. Specifically,FIG. 3shows a detailed version of steps215-225according to one or more embodiments.

The flowchart begins at305and the image processing module155identifies a first subset of the image data associated with a region of interest. As described above, the region of interest may be identified in any number of ways, and the determination may be based on user input, sensor data, default settings, and the like.

The flowchart continues at310, and the image processing module155identifies a second subset of the image data. The second subset of image data may be different than the first subset of image data. According to one or more embodiments, the second subset of data may include the remainder of the image data for a frame that does not include the first subset of image data. In an alternative embodiment, the second subset of image data may include the entirety of the image data for a frame, including the first subset of image data.

The flowchart continues at315and the first subset of image data is processed by a first processing pipeline. In one or more embodiments, the first processing pipeline produces a high quality image data, and may be a higher quality image data than that produced by a second processing pipeline. For example, in one or more embodiments, the first processing pipeline produces greater pixels per degree (PPD) than the second processing pipeline. The first processing pipeline may render image data in a higher quality than a second processing pipeline in a number of way. For example, the first processing pipeline may render a higher quality image by utilizing a more aggressive noise processing, tone mapping, color rendition, desaturation, and the like.

At320, the second subset of image data is processed by a second processing pipeline, which may produce a lower quality image than the first quality pipeline. In one or more embodiments, the second processing pipeline may include a different processing method than that of the first processing pipeline. In one or more embodiments, processing the second subset of data by a second processing pipeline may include at325, downsampling the second subset of image data such that only a portion of the second subset of image data is considered. In one or more embodiments, downsampling reduces the pixels used to represent the images by only considering a portion of the captured pixels. Then, at330, the downsampled second subset of data (i.e., the portion of the second subset of image data) is processed using the first processing pipeline.

The flowchart continues at335, and the image processing module155combines the first processed image data and the second processed image data to obtain a processed image frame. That is, the image processing module155may combine the processed first subset of image data and the processed second subset of image data to obtain a processed image frame. In the instance where the second subset of image data comprises the entirety of image data for the frame, the portion of the second subset of image data corresponding to first subset of image data may be replaced by the first subset of image data.

Although the above description includes two data flows, in one or more embodiments, the image processing module155may generate the image frames for the video feed with additional data flows. For example, the image processing pipeline may utilize a third data flow such that image data corresponding to a region of interest is processed at a highest quality level, a region surrounding the region of interest is processed at a medium quality level, and a region surrounding the medium quality level region is processed at a lowest quality level, thus providing an extra layer of granularity to the foveation. In this example, the outermost region away from the region of interest may be downsampled the most aggressively, for example.

FIG. 4shows, in flow diagram form, an example of generating a foveated image frame in accordance with one or more embodiments. The flow begins at405, and the image processing module155identifies image data associated with a particular image frame to be processed. The image processing module155identifies a portion of the image data associated with a region of interest to obtain a first subset of image data. As described above, the portion of the image data may include image data that includes the region of interest. The identified portion of image data is, as shown, first image data410. Although the first image data comprising the image data is depicted as a rectangular portion, the shape of the first image data comprising the region of interest may be an alternative geometric shape, such as a hexagon. Further, the shape of the first image data comprising the region of interest may be a regular or irregular shape. The image processing module155also identifies second image data425, which is a second subset of the image data405. In one or more embodiments, the second image data425may be different than the first image data410. In one embodiment, the second image data425may include the image data405except for the first image data410. In another embodiment, the second image data425may include all of the image data405, including the first image data410.

With respect to the first image data410, the flow diagram400continues at image processing pipeline A420. In one or more embodiments, image processing pipeline A420is configured to process image data at a high image quality. The result of the image processing pipeline A420processing the first image data410is first processed image data. The first image processing pipeline A420may be a first signal processing pipeline

With respect to the second image data425the image processing module155directs the second image data425to be processed by image processing pipeline B430. In one or more embodiments, the second image data425may be downsampled prior to being processed by image processing pipeline B430. As an example, only every fourth pixel captured by the camera110may be processed. According to one or more embodiments, image processing pipeline B430is configured to process image data at a lower quality and/or in a more efficient manner than image processing pipeline A420. The result of image processing pipeline B430processing second image data425, is second processed image data.

After the first image data410is processed by image processing pipeline A420and second image data425is processed by image processing pipeline B430, then the first processed image data from image processing pipeline A420and the second processed image data from image processing pipeline B430is combined to form composite frame435. If the second image data425was downsampled prior to being processed by image processing pipeline B430, then the image processing module155may be upscaled prior to being combined with the first processed image data to obtain the composite frame435. For example, the image processing module155may paste the first processed image data into the proper location of the second processed image data to obtain a composite frame435, resulting in a foveated image. As another example, combining the first image data and the second image data may include blending the first image data and the second image data together to form a smooth transition between the boundary of the first image data and the second image data. Thus, between a region of the first image data and a region of the second image data may be a transition region across multiple pixels. As described above, the flow diagram400may be repeated with the next image data for a next frame of a video feed until the video feed ceases.

FIG. 5shows, in flow diagram form, another example of generating a foveated image frame in accordance with one or more embodiments. The example flow diagram500includes many of the same components as described above with respect toFIG. 4. However, the example flow diagram500is intended to depict an alternative embodiment. In one or more embodiments, the various components and actions described withinFIG. 4andFIG. 5may be optionally combined or reorganized to form additional embodiments. Thus, the particular flow diagram500is shown merely as an example.

The flow begins at405, and the image processing module155identifies image data associated with a particular image frame to be processed. As described above, the image processing module155identifies a portion of the image data associated with a region of interest to obtain a first subset of image data. As described above, the portion of the image data may include image data that includes the region of interest. The identified portion of image data is, as shown, first image data410. The image processing module155also identifies second image data425, which is a second subset of the image data405. In one or more embodiments, the second image data425may be different than the first image data410. In one embodiment, the second image data425may include the image data405except for the first image data410. As shown, the second image data425may include all of the image data405, including the first image data410. Thus, the first image data410may be a copy of a cropped portion of the image data405.

With respect to the first image data410, the flow diagram400continues at image processing pipeline A420. In one or more embodiments, image processing pipeline A420is configured to process image data at a high image quality. The result of the image processing pipeline A420processing the first image data410is first processed image data.

Distinct from flow400, in flow500, respect to the second image data425the image processing module155may downsample the image data to obtain downsampled image data515. In one or more embodiments, the image processing module155may obtain the downsampled image data515by taking a subset of the second image data425. The flow continues by the image processing module155directing the downsampled image data515to be processed by image processing pipeline A420. Thus, the same pipeline may be used for the first image data410and the downsampled image data515, but because the downsampled image data515includes only a portion of the second image data425, the downsampled image data515is processed by image processing pipeline420more efficiently than if the second image data425were processed. The result of image processing pipeline A420processing the downsampled image data515, is second processed image data.

After the downsampled image data515is processed by image processing pipeline A420, then the processed image data is upscaled to obtain upscaled image data520. In one or more embodiments, upscaling the processed downsampled image data results in the second processed image data to return to the same scale as the first processed image data. After the first image data410is processed by image processing pipeline A420and second processed image data is upscaled, then the first processed image data and the upscaled second processed image data are combined to form composite frame435. In one or more embodiments, the image processing module155determines the region of interest boundaries with respect to the raw image data for the frame, and combines the two sets of processed image data to revert back to the native grid. As described above, the flow diagram500may be repeated with the next image data for a next frame of a video feed until the video feed ceases.

Referring now toFIG. 6, a simplified functional block diagram of illustrative multifunction device600is shown according to one embodiment. Multifunction electronic device600may include processor605, display610, user interface615, graphics hardware620, device sensors625(e.g., proximity sensor/ambient light sensor, accelerometer and/or gyroscope), microphone630, audio codec(s)635, speaker(s)640, communications circuitry645, digital image capture circuitry650(e.g., including camera system100) video codec(s)655(e.g., in support of digital image capture unit650), memory660, storage device665, and communications bus670. Multifunction electronic device600may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), personal music player, mobile telephone, or a tablet computer.

Processor605may execute instructions necessary to carry out or control the operation of many functions performed by device600(e.g., such as the generation and/or processing of images and single and multi-camera calibration as disclosed herein). Processor605may, for instance, drive display610and receive user input from user interface615. User interface615may allow a user to interact with device600. For example, user interface615can take a variety of forms, such as a button, keypad, dial, a click wheel, keyboard, display screen and/or a touch screen. Processor605may also, for example, be a system-on-chip such as those found in mobile devices and include a dedicated graphics processing unit (GPU). Processor605may be based on reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architectures or any other suitable architecture and may include one or more processing cores. Graphics hardware620may be special purpose computational hardware for processing graphics and/or assisting processor605to process graphics information. In one embodiment, graphics hardware620may include a programmable GPU.

Image capture circuitry650may include lens assembly680associated with sensor element690. Image capture circuitry650may capture still and/or video images. Output from image capture circuitry650may be processed, at least in part, by video codec(s)655and/or processor605and/or graphics hardware620, and/or a dedicated image processing unit or pipeline incorporated within circuitry665. Images so captured may be stored in memory660and/or storage655.

Sensor and camera circuitry650may capture still and video images that may be processed in accordance with this disclosure, at least in part, by video codec(s)655and/or processor605and/or graphics hardware620, and/or a dedicated image processing unit incorporated within circuitry650. Images so captured may be stored in memory660and/or storage665. Memory660may include one or more different types of media used by processor605and graphics hardware620to perform device functions. For example, memory660may include memory cache, read-only memory (ROM), and/or random access memory (RAM). Storage665may store media (e.g., audio, image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage665may include one more non-transitory computer readable storage mediums including, for example, magnetic disks (fixed, floppy, and removable) and tape, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory660and storage665may be used to tangibly retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor605such computer program code may implement one or more of the methods described herein.

The scope of the disclosed subject matter therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”