Patent ID: 12197266

DETAILED DESCRIPTION

This document includes disclosure of systems, apparatus, and methods for dynamic power allocation for memory using multiple interleaving patterns. Power consumption is an important consideration in the design of image capture devices (e.g., cameras). Record time, battery life and capture at low temperature are key issues that may be impacted by a power consumption profile for an image capture device. Having multiple memory devices (e.g., double data rate synchronous dynamic random access memory (DDRs)) creates an opportunity to fine tune power consumption to different use cases by selectively powering down (e.g., turning off or disabling) a subset of the available memory devices when they are not need for a current use case. It is also preferable to be able to change the memory usage mode dynamically during use, without requiring a reboot of the image capture device that could cause delay and inconvenience to a user.

Some implementations described herein turn off a subset of available DDRs in the main system-on-a-chip (SOC) of a camera to save power while meeting the performance requirements of a current use case for the camera. For example, an image capture device with four DDRs running may consume 400 mW more than the same use case running with only two of the DDRs. There may be little for a user gain to by running with four DDRs compared to two, except that four DDRs may be required for some higher performance use cases and it may be complex to turn on/turn off a DDR. Some implementations use hardware and/or firmware to dynamically switch DDR(s) on and off according to use case needs.

Dynamically turning DDRs on and off may be enabled by organizing a memory map with DDR address interleaving in particular ways to facilitate seamless switching between modes and avoid reboot the device using the DDRs. For example, a memory management circuitry (e.g., a memory management unit (MMU) in the device may implement multiple interleaving patterns that can be selected based on the mode (e.g., performance mode or power conservation mode) to translate virtual addresses into physical addresses in the DDRs. The different interleaving patterns may have a common portion corresponding to a range of virtual addresses that can be used to store instructions (e.g., instructions of OS software) so that execution can continue seamlessly through mode transitions that turn a subset of the available DDRs on or off. The performance mode interleaving pattern may implement interleaving using all available DDRs in range of virtual addresses used for high heap data (e.g., pixel buffers) to increase memory bandwidth in the performance mode. Thus, from a software perspective, a change in mode may be experienced as simply a change in heap size and a change in memory bandwidth.

Image capture devices implementing these techniques for dynamic power allocation for memory may have advantages, such as, for example, reducing power consumption of the image capture device, while tailoring memory usage to the requirements of different use cases for memory capacity and memory bandwidth.

FIGS.1A-Bare isometric views of an example of an image capture device100. The image capture device100may include a body102, a lens104structured on a front surface of the body102, various indicators on the front surface of the body102(such as light-emitting diodes (LEDs), displays, and the like), various input mechanisms (such as buttons, switches, and/or touch-screens), and electronics (such as imaging electronics, power electronics, etc.) internal to the body102for capturing images via the lens104and/or performing other functions. The lens104is configured to receive light incident upon the lens104and to direct received light onto an image sensor internal to the body102. The image capture device100may be configured to capture images and video and to store captured images and video for subsequent display or playback.

The image capture device100may include an LED or another form of indicator106to indicate a status of the image capture device100and a liquid-crystal display (LCD) or other form of a display108to show status information such as battery life, camera mode, elapsed time, and the like. The image capture device100may also include a mode button110and a shutter button112that are configured to allow a user of the image capture device100to interact with the image capture device100. For example, the mode button110and the shutter button112may be used to turn the image capture device100on and off, scroll through modes and settings, and select modes and change settings. The image capture device100may include additional buttons or interfaces (not shown) to support and/or control additional functionality.

The image capture device100may include a door114coupled to the body102, for example, using a hinge mechanism116. The door114may be secured to the body102using a latch mechanism118that releasably engages the body102at a position generally opposite the hinge mechanism116. The door114may also include a seal120and a battery interface122. When the door114is an open position, access is provided to an input-output (I/O) interface124for connecting to or communicating with external devices as described below and to a battery receptacle126for placement and replacement of a battery (not shown). The battery receptacle126includes operative connections (not shown) for power transfer between the battery and the image capture device100. When the door114is in a closed position, the seal120engages a flange (not shown) or other interface to provide an environmental seal, and the battery interface122engages the battery to secure the battery in the battery receptacle126. The door114can also have a removed position (not shown) where the entire door114is separated from the image capture device100, that is, where both the hinge mechanism116and the latch mechanism118are decoupled from the body102to allow the door114to be removed from the image capture device100.

The image capture device100may include a microphone128on a front surface and another microphone130on a side surface. The image capture device100may include other microphones on other surfaces (not shown). The microphones128,130may be configured to receive and record audio signals in conjunction with recording video or separate from recording of video. The image capture device100may include a speaker132on a bottom surface of the image capture device100. The image capture device100may include other speakers on other surfaces (not shown). The speaker132may be configured to play back recorded audio or emit sounds associated with notifications.

A front surface of the image capture device100may include a drainage channel134. A bottom surface of the image capture device100may include an interconnect mechanism136for connecting the image capture device100to a handle grip or other securing device. In the example shown inFIG.1B, the interconnect mechanism136includes folding protrusions configured to move between a nested or collapsed position as shown and an extended or open position (not shown) that facilitates coupling of the protrusions to mating protrusions of other devices such as handle grips, mounts, clips, or like devices.

The image capture device100may include an interactive display138that allows for interaction with the image capture device100while simultaneously displaying information on a surface of the image capture device100.

The image capture device100ofFIGS.1A-Bincludes an exterior that encompasses and protects internal electronics. In the present example, the exterior includes six surfaces (i.e., a front face, a left face, a right face, a back face, a top face, and a bottom face) that form a rectangular cuboid. Furthermore, both the front and rear surfaces of the image capture device100are rectangular. In other embodiments, the exterior may have a different shape. The image capture device100may be made of a rigid material such as plastic, aluminum, steel, or fiberglass. The image capture device100may include features other than those described here. For example, the image capture device100may include additional buttons or different interface features, such as interchangeable lenses, cold shoes, and hot shoes that can add functional features to the image capture device100.

The image capture device100may include various types of image sensors, such as charge-coupled device (CCD) sensors, active pixel sensors (APS), complementary metal-oxide-semiconductor (CMOS) sensors, N-type metal-oxide-semiconductor (NMOS) sensors, and/or any other image sensor or combination of image sensors.

Although not illustrated, in various embodiments, the image capture device100may include other additional electrical components (e.g., an image processor, camera system-on-chip (SoC), etc.), which may be included on one or more circuit boards within the body102of the image capture device100.

The image capture device100may interface with or communicate with an external device, such as an external user interface device (not shown), via a wired or wireless computing communication link (e.g., the I/O interface124). Any number of computing communication links may be used. The computing communication link may be a direct computing communication link or an indirect computing communication link, such as a link including another device or a network, such as the internet, may be used.

In some implementations, the computing communication link may be a Wi-Fi link, an infrared link, a Bluetooth (BT) link, a cellular link, a ZigBee link, a near field communications (NFC) link, such as an ISO/IEC 20643 protocol link, an Advanced Network Technology interoperability (ANT+) link, and/or any other wireless communications link or combination of links.

In some implementations, the computing communication link may be an HDMI link, a USB link, a digital video interface link, a display port interface link, such as a Video Electronics Standards Association (VESA) digital display interface link, an Ethernet link, a Thunderbolt link, and/or other wired computing communication link.

The image capture device100may transmit images, such as panoramic images, or portions thereof, to the external user interface device via the computing communication link, and the external user interface device may store, process, display, or a combination thereof the panoramic images.

The external user interface device may be a computing device, such as a smartphone, a tablet computer, a phablet, a smart watch, a portable computer, personal computing device, and/or another device or combination of devices configured to receive user input, communicate information with the image capture device100via the computing communication link, or receive user input and communicate information with the image capture device100via the computing communication link.

The external user interface device may display, or otherwise present, content, such as images or video, acquired by the image capture device100. For example, a display of the external user interface device may be a viewport into the three-dimensional space represented by the panoramic images or video captured or created by the image capture device100.

The external user interface device may communicate information, such as metadata, to the image capture device100. For example, the external user interface device may send orientation information of the external user interface device with respect to a defined coordinate system to the image capture device100, such that the image capture device100may determine an orientation of the external user interface device relative to the image capture device100.

Based on the determined orientation, the image capture device100may identify a portion of the panoramic images or video captured by the image capture device100for the image capture device100to send to the external user interface device for presentation as the viewport. In some implementations, based on the determined orientation, the image capture device100may determine the location of the external user interface device and/or the dimensions for viewing of a portion of the panoramic images or video.

The external user interface device may implement or execute one or more applications to manage or control the image capture device100. For example, the external user interface device may include an application for controlling camera configuration, video acquisition, video display, or any other configurable or controllable aspect of the image capture device100.

The user interface device, such as via an application, may generate and share, such as via a cloud-based or social media service, one or more images, or short video clips, such as in response to user input. In some implementations, the external user interface device, such as via an application, may remotely control the image capture device100such as in response to user input.

The external user interface device, such as via an application, may display unprocessed or minimally processed images or video captured by the image capture device100contemporaneously with capturing the images or video by the image capture device100, such as for shot framing or live preview, and which may be performed in response to user input. In some implementations, the external user interface device, such as via an application, may mark one or more key moments contemporaneously with capturing the images or video by the image capture device100, such as with a tag or highlight in response to a user input or user gesture.

The external user interface device, such as via an application, may display or otherwise present marks or tags associated with images or video, such as in response to user input. For example, marks may be presented in a camera roll application for location review and/or playback of video highlights.

The external user interface device, such as via an application, may wirelessly control camera software, hardware, or both. For example, the external user interface device may include a web-based graphical interface accessible by a user for selecting a live or previously recorded video stream from the image capture device100for display on the external user interface device.

The external user interface device may receive information indicating a user setting, such as an image resolution setting (e.g., 3840 pixels by 2160 pixels), a frame rate setting (e.g., 60 frames per second (fps)), a location setting, and/or a context setting, which may indicate an activity, such as mountain biking, in response to user input, and may communicate the settings, or related information, to the image capture device100.

The image capture device100may be used to implement some or all of the techniques described in this disclosure, such as the technique700described inFIG.7.

FIGS.2A-Billustrate another example of an image capture device200. The image capture device200includes a body202and two camera lenses204and206disposed on opposing surfaces of the body202, for example, in a back-to-back configuration, Janus configuration, or offset Janus configuration. The body202of the image capture device200may be made of a rigid material such as plastic, aluminum, steel, or fiberglass.

The image capture device200includes various indicators on the front of the surface of the body202(such as LEDs, displays, and the like), various input mechanisms (such as buttons, switches, and touch-screen mechanisms), and electronics (e.g., imaging electronics, power electronics, etc.) internal to the body202that are configured to support image capture via the two camera lenses204and206and/or perform other imaging functions.

The image capture device200includes various indicators, for example, LEDs208,210to indicate a status of the image capture device100. The image capture device200may include a mode button212and a shutter button214configured to allow a user of the image capture device200to interact with the image capture device200, to turn the image capture device200on, and to otherwise configure the operating mode of the image capture device200. It should be appreciated, however, that, in alternate embodiments, the image capture device200may include additional buttons or inputs to support and/or control additional functionality.

The image capture device200may include an interconnect mechanism216for connecting the image capture device200to a handle grip or other securing device. In the example shown inFIGS.2A and2B, the interconnect mechanism216includes folding protrusions configured to move between a nested or collapsed position (not shown) and an extended or open position as shown that facilitates coupling of the protrusions to mating protrusions of other devices such as handle grips, mounts, clips, or like devices.

The image capture device200may include audio components218,220,222such as microphones configured to receive and record audio signals (e.g., voice or other audio commands) in conjunction with recording video. The audio component218,220,222can also be configured to play back audio signals or provide notifications or alerts, for example, using speakers. Placement of the audio components218,220,222may be on one or more of several surfaces of the image capture device200. In the example ofFIGS.2A and2B, the image capture device200includes three audio components218,220,222, with the audio component218on a front surface, the audio component220on a side surface, and the audio component222on a back surface of the image capture device200. Other numbers and configurations for the audio components are also possible.

The image capture device200may include an interactive display224that allows for interaction with the image capture device200while simultaneously displaying information on a surface of the image capture device200. The interactive display224may include an I/O interface, receive touch inputs, display image information during video capture, and/or provide status information to a user. The status information provided by the interactive display224may include battery power level, memory card capacity, time elapsed for a recorded video, etc.

The image capture device200may include a release mechanism225that receives a user input to in order to change a position of a door (not shown) of the image capture device200. The release mechanism225may be used to open the door (not shown) in order to access a battery, a battery receptacle, an I/O interface, a memory card interface, etc. (not shown) that are similar to components described in respect to the image capture device100ofFIGS.1A and1B.

In some embodiments, the image capture device200described herein includes features other than those described. For example, instead of the I/O interface and the interactive display224, the image capture device200may include additional interfaces or different interface features. For example, the image capture device200may include additional buttons or different interface features, such as interchangeable lenses, cold shoes, and hot shoes that can add functional features to the image capture device200.

FIG.2Cis a top view of the image capture device200ofFIGS.2A-BandFIG.2Dis a partial cross-sectional view of the image capture device200ofFIG.2C. The image capture device200is configured to capture spherical images, and accordingly, includes a first image capture device226and a second image capture device228. The first image capture device226defines a first field-of-view230and includes the lens204that receives and directs light onto a first image sensor232. Similarly, the second image capture device228defines a second field-of-view234and includes the lens206that receives and directs light onto a second image sensor236. To facilitate the capture of spherical images, the image capture devices226and228(and related components) may be arranged in a back-to-back (Janus) configuration such that the lenses204,206face in generally opposite directions.

The fields-of-view230,234of the lenses204,206are shown above and below boundaries238,240indicated in dotted line. Behind the first lens204, the first image sensor232may capture a first hyper-hemispherical image plane from light entering the first lens204, and behind the second lens206, the second image sensor236may capture a second hyper-hemispherical image plane from light entering the second lens206.

One or more areas, such as blind spots242,244may be outside of the fields-of-view230,234of the lenses204,206so as to define a “dead zone.” In the dead zone, light may be obscured from the lenses204,206and the corresponding image sensors232,236, and content in the blind spots242,244may be omitted from capture. In some implementations, the image capture devices226,228may be configured to minimize the blind spots242,244.

The fields-of-view230,234may overlap. Stitch points246,248proximal to the image capture device200, that is, locations at which the fields-of-view230,234overlap, may be referred to herein as overlap points or stitch points. Content captured by the respective lenses204,206that is distal to the stitch points246,248may overlap.

Images contemporaneously captured by the respective image sensors232,236may be combined to form a combined image. Generating a combined image may include correlating the overlapping regions captured by the respective image sensors232,236, aligning the captured fields-of-view230,234, and stitching the images together to form a cohesive combined image.

A slight change in the alignment, such as position and/or tilt, of the lenses204,206, the image sensors232,236, or both, may change the relative positions of their respective fields-of-view230,234and the locations of the stitch points246,248. A change in alignment may affect the size of the blind spots242,244, which may include changing the size of the blind spots242,244unequally.

Incomplete or inaccurate information indicating the alignment of the image capture devices226,228, such as the locations of the stitch points246,248, may decrease the accuracy, efficiency, or both of generating a combined image. In some implementations, the image capture device200may maintain information indicating the location and orientation of the lenses204,206and the image sensors232,236such that the fields-of-view230,234, the stitch points246,248, or both may be accurately determined; the maintained information may improve the accuracy, efficiency, or both of generating a combined image.

The lenses204,206may be laterally offset from each other, may be off-center from a central axis of the image capture device200, or may be laterally offset and off-center from the central axis. As compared to image capture devices with back-to-back lenses, such as lenses aligned along the same axis, image capture devices including laterally offset lenses may include substantially reduced thickness relative to the lengths of the lens barrels securing the lenses. For example, the overall thickness of the image capture device200may be close to the length of a single lens barrel as opposed to twice the length of a single lens barrel as in a back-to-back lens configuration. Reducing the lateral distance between the lenses204,206may improve the overlap in the fields-of-view230,234. In another embodiment (not shown), the lenses204,206may be aligned along a common imaging axis.

Images or frames captured by the image capture devices226,228may be combined, merged, or stitched together to produce a combined image, such as a spherical or panoramic image, which may be an equirectangular planar image. In some implementations, generating a combined image may include use of techniques including noise reduction, tone mapping, white balancing, or other image correction. In some implementations, pixels along the stitch boundary may be matched accurately to minimize boundary discontinuities.

The image capture device200may be used to implement some or all of the techniques described in this disclosure, such as the technique700described inFIG.7.

FIG.3is a block diagram of electronic components in an image capture device300. The image capture device300may be a single-lens image capture device, a multi-lens image capture device, or variations thereof, including an image capture device with multiple capabilities such as use of interchangeable integrated sensor lens assemblies. The description of the image capture device300is also applicable to the image capture devices100,200ofFIGS.1A-Band2A-D.

The image capture device300includes a body302which includes electronic components such as capture components310, a processing apparatus320, data interface components330, movement sensors340, power components350, and/or user interface components360.

The capture components310include one or more image sensors312for capturing images and one or more microphones314for capturing audio.

The image sensor(s)312is configured to detect light of a certain spectrum (e.g., the visible spectrum or the infrared spectrum) and convey information constituting an image as electrical signals (e.g., analog or digital signals). The image sensor(s)312detects light incident through a lens coupled or connected to the body302. The image sensor(s)312may be any suitable type of image sensor, such as a charge-coupled device (CCD) sensor, active pixel sensor (APS), complementary metal-oxide-semiconductor (CMOS) sensor, N-type metal-oxide-semiconductor (NMOS) sensor, and/or any other image sensor or combination of image sensors. Image signals from the image sensor(s)312may be passed to other electronic components of the image capture device300via a bus380, such as to the processing apparatus320. In some implementations, the image sensor(s)312includes a digital-to-analog converter. A multi-lens variation of the image capture device300can include multiple image sensors312.

The microphone(s)314is configured to detect sound, which may be recorded in conjunction with capturing images to form a video. The microphone(s)314may also detect sound in order to receive audible commands to control the image capture device300.

The processing apparatus320may be configured to perform image signal processing (e.g., filtering, tone mapping, stitching, and/or encoding) to generate output images based on image data from the image sensor(s)312. The processing apparatus320may include one or more processors having single or multiple processing cores. In some implementations, the processing apparatus320may include an application specific integrated circuit (ASIC). For example, the processing apparatus320may include a custom image signal processor. The processing apparatus320may exchange data (e.g., image data) with other components of the image capture device300, such as the image sensor(s)312, via the bus380.

The processing apparatus320may include memory, such as a random-access memory (RAM) device, flash memory, or another suitable type of storage device, such as a non-transitory computer-readable memory. The memory of the processing apparatus320may include executable instructions and data that can be accessed by one or more processors of the processing apparatus320. For example, the processing apparatus320may include one or more dynamic random-access memory (DRAM) modules, such as double data rate synchronous dynamic random-access memory (DDR SDRAM). In some implementations, the processing apparatus320may include a digital signal processor (DSP). More than one processing apparatus may also be present or associated with the image capture device300. For example, the processing apparatus320may include the processing apparatus400ofFIG.4.

The data interface components330enable communication between the image capture device300and other electronic devices, such as a remote control, a smartphone, a tablet computer, a laptop computer, a desktop computer, or a storage device. For example, the data interface components330may be used to receive commands to operate the image capture device300, transfer image data to other electronic devices, and/or transfer other signals or information to and from the image capture device300. The data interface components330may be configured for wired and/or wireless communication. For example, the data interface components330may include an I/O interface332that provides wired communication for the image capture device, which may be a USB interface (e.g., USB type-C), a high-definition multimedia interface (HDMI), or a FireWire interface. The data interface components330may include a wireless data interface334that provides wireless communication for the image capture device300, such as a Bluetooth interface, a ZigBee interface, and/or a Wi-Fi interface. The data interface components330may include a storage interface336, such as a memory card slot configured to receive and operatively couple to a storage device (e.g., a memory card) for data transfer with the image capture device300(e.g., for storing captured images and/or recorded audio and video).

The movement sensors340may detect the position and movement of the image capture device300. The movement sensors340may include a position sensor342, an accelerometer344, or a gyroscope346. The position sensor342, such as a global positioning system (GPS) sensor, is used to determine a position of the image capture device300. The accelerometer344, such as a three-axis accelerometer, measures linear motion (e.g., linear acceleration) of the image capture device300. The gyroscope346, such as a three-axis gyroscope, measures rotational motion (e.g., rate of rotation) of the image capture device300. Other types of movement sensors340may also be present or associated with the image capture device300.

The power components350may receive, store, and/or provide power for operating the image capture device300. The power components350may include a battery interface352and a battery354. The battery interface352operatively couples to the battery354, for example, with conductive contacts to transfer power from the battery354to the other electronic components of the image capture device300. The power components350may also include an external interface356, and the power components350may, via the external interface356, receive power from an external source, such as a wall plug or external battery, for operating the image capture device300and/or charging the battery354of the image capture device300. In some implementations, the external interface356may be the I/O interface332. In such an implementation, the I/O interface332may enable the power components350to receive power from an external source over a wired data interface component (e.g., a USB type-C cable).

The user interface components360may allow the user to interact with the image capture device300, for example, providing outputs to the user and receiving inputs from the user. The user interface components360may include visual output components362to visually communicate information and/or present captured images to the user. The visual output components362may include one or more lights364and/or more displays366. The display(s)366may be configured as a touch screen that receives inputs from the user. The user interface components360may also include one or more speakers368. The speaker(s)368can function as an audio output component that audibly communicates information and/or presents recorded audio to the user. The user interface components360may also include one or more physical input interfaces370that are physically manipulated by the user to provide input to the image capture device300. The physical input interfaces370may, for example, be configured as buttons, toggles, or switches. The user interface components360may also be considered to include the microphone(s)314, as indicated in dotted line, and the microphone(s)314may function to receive audio inputs from the user, such as voice commands.

The image capture device300may be used to implement some or all of the techniques described in this disclosure, such as the technique700described inFIG.7or the technique800described inFIG.8.

FIG.4is a block diagram of an example of a processing apparatus400. The processing apparatus400includes a set of memory devices410and a memory management circuitry430configured to translate virtual addresses into physical addresses of memory locations in the set of memory devices410. The processing apparatus400includes a power conservation circuitry440configured to selectively power down memory devices in the set of memory devices410. The processing apparatus400includes a non-volatile memory450storing software that may be loaded into the set of memory devices410and executed by the processing apparatus400. Although not explicitly shown inFIG.1, the processing apparatus400may also include one or more processor cores that are configured to fetch and execute instructions from the set of memory devices410and to load data from and store data to the set of memory devices410. For example, the processing apparatus400may be part of an image capture device (e.g., the image capture device100) and may be used to capture images. For example, the processing apparatus400may be used to implement the technique700ofFIG.7. For example, the processing apparatus400may be used to implement the technique800ofFIG.8.

The processing apparatus400includes a set of memory devices410, including a first subset420of one or more memory devices and a second subset422of one or more memory devices that is disjoint from the first subset. In this example, the first subset420includes a memory device412and a memory device414, but the first subset420may include any positive number of memory devices. In this example, the second subset422includes a memory device416and a memory device418, but the second subset422may include any positive number of memory devices. In some implementations, the set of memory devices410is partitioned into just the first subset420and the second subset422. In some implementations, the set of memory devices410may be partitioned into more than two subsets, which may enable more levels of power conservation by selectively powering down one or more subsets of memory devices in the set of memory devices410. In some implementations, each memory device in the set of memory devices410is a memory bank that can be accessed in parallel with other memory banks in the set of memory devices410, which may enable interleaving patterns to be used to increase the speed at which data may be accessed in the set of memory devices410. For example, a memory device (e.g., the memory device412, the memory device414, the memory device416, and/or the memory device418) in the set of memory devices410may be a double data rate synchronous dynamic random access memory chip (a DDR chip). In some implementations, a memory device (e.g., the memory device412, the memory device414, the memory device416, and/or the memory device418) in the set of memory devices410may be integrated in a system on a chip (SOC) with other components of the processing apparatus400. In some implementations, a memory device (e.g., the memory device412, the memory device414, the memory device416, and/or the memory device418) in the set of memory devices410may be integrated with other components of the processing apparatus400in a package containing and connecting multiple integrated circuit chips (e.g., stacked in a Package on a Package (PoP) configuration). In some implementations, a memory device (e.g., the memory device412, the memory device414, the memory device416, and/or the memory device418) in the set of memory devices410may be discrete devices connected with other components of the processing apparatus400via conductors (e.g., traces on a printed circuit board (PCB) or a ribbon cable). For example, the set of memory devices410may be the set of memory devices500ofFIG.5.

The processing apparatus400includes a memory management circuitry430(e.g., a memory management unit (MMU)). The memory management circuitry430is configured to translate virtual addresses into physical addresses of memory locations in the set of memory devices410using a first interleaving pattern when operating in a first mode and translate virtual addresses using a second interleaving pattern when operating in a second mode. The first interleaving pattern and the second interleaving pattern both map virtual addresses in a first range exclusively to memory devices in the first subset420. The first interleaving pattern maps virtual addresses in a second range to memory devices in the first subset420and in the second subset422, which may increase memory bandwidth for access to data stored at virtual addresses in the second range by interleaving the data across more memory devices. The second interleaving pattern maps virtual addresses in the second range exclusively to memory devices in the first subset420, which may enable memory devices in the second subset422to be powered down while in the second mode to reduce power consumption at the cost of reducing memory bandwidth for access to data stored at virtual addresses in the second range. For example, the memory management circuitry430may be configured to dynamically change between the first mode and the second mode without rebooting the processing apparatus400running software stored in the set of memory devices410. For example, memory management circuitry430may dynamically change between the first mode and the second mode in response to a use case indication received for a device (e.g., the image capture device100) including the processing apparatus400. Some use cases may require more memory capacity and/or memory bandwidth than other use cases. Dynamic switching between the first mode and the second mode may enable optimization for power consumption by enable the selective powering down of memory devices in the set of memory devices410. This dynamic changing between the first mode and the second mode may be enabled by the mapping of virtual addresses in the first range being unchanged between the first interleaving pattern and the second interleaving pattern, so that the first range may be used to store instructions for running software across transitions between the first mode and the second mode. Less memory may be available for use when in the second mode. For example, the memory management circuitry430may be configured to, when operating in the second mode, return a memory fault (e.g., return a bus error) for virtual addresses outside of the first range and the second range. In some implementations, the first interleaving pattern maps virtual addresses in a third range exclusively to memory devices in the second subset422, and the second interleaving pattern maps virtual addresses in the third range to a memory fault. In some implementations, the first interleaving pattern maps virtual addresses in a fourth range to memory devices in the first subset420and in the second subset422, and the second interleaving pattern maps virtual addresses in the fourth range to a memory fault. For example, the memory management circuitry430may be integrated with a processor core of the processing apparatus400(e.g., coupled with an L1 cache). The first interleaving pattern and the second interleaving pattern may be encoded in various ways by the memory management circuitry430. For example, the first interleaving pattern and the second interleaving pattern may be encoded by alternative page tables stored by the processing apparatus400. In some implementations, the first interleaving pattern uses a 512-byte page size with consecutive pages in virtual memory mapped to different memory devices of the set of memory devices410. For example, the first interleaving pattern and the second interleaving pattern may be encoded by logic circuitry (e.g., gates) directly implementing the two address translation mappings. For example, the first interleaving pattern may be the interleaving pattern600ofFIG.6A. For example, the second interleaving pattern may be the interleaving pattern650ofFIG.6B.

The processing apparatus400includes a power conservation circuitry440configured to power down memory devices in the second subset422while the memory management circuitry is operating in the second mode. For example, the power conservation circuitry440may power down the memory devices in the second subset422by changing a voltage on an enable conductor of the memory devices in the second subset422. For example, the power conservation circuitry440may power down the memory devices in the second subset422by gating a clock signal into the memory devices in the second subset422. In some implementations, the power conservation circuitry440is integrated with the memory management circuitry430. In some implementations, the power conservation circuitry440is separate from the memory management circuitry430.

The processing apparatus400includes a non-volatile memory450storing software that is configured to store heap data at virtual addresses in the second range. In some implementations, the non-volatile memory450stores software that is configured to store operating system code at virtual addresses in the first range.

FIG.5is a block diagram of an example of a set of memory devices500used by a processing apparatus. The set of memory devices500includes a first memory device502named DDR 0, a second memory device504named DDR 1, a third memory device506named DDR 2, and a fourth memory device508named DDR 3. The set of memory devices500may be partitioned into a first subset, including the first memory device502and the second memory device504, and a second subset, including the third memory device506and the fourth memory device508. The individual memory devices may themselves be partitioned into regions corresponding to different ranges of physical address in the memory device. Some of the regions of a memory device may be statically configured in the sense that a processing apparatus accessing them always maps physical addresses in statically configured regions to the same virtual addresses. Other regions of a memory device may be dynamically configured in the sense that a processing apparatus accessing them maps physical addresses in dynamically configured regions to different virtual addresses depending on a current mode selection. Note that the configuration of this address mapping may be implemented outside of the set of memory devices themselves, such as in a memory management circuitry (e.g., the memory management circuitry430).

The first memory device502includes a first region510is that statically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aand the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does not change between the available interleaving patterns used in different modes. The first memory device502includes a second region512is dynamically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aor the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does change between the available interleaving patterns used in different modes. For example, the modes (e.g., a high-performance mode and a power conservation mode) that control how the second region512is addressed may be changed responsive to an indication of a use case change for device (e.g., the image capture device100or the image capture device200) that includes the set of memory devices500. Some use cases may require more memory capacity and/or memory bandwidth than other use cases.

Similarly, the second memory device504includes a first region520that is statically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aand the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does not change between the available interleaving patterns used in different modes. The second memory device504includes a second region522is dynamically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aor the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does change between the available interleaving patterns used in different modes. For example, the modes (e.g., a high-performance mode and a power conservation mode) that control how the second region522is addressed may be changed responsive to an indication of a use case change for device (e.g., the image capture device100or the image capture device200) that includes the set of memory devices500. The first memory device502and the second memory device504may constitute a first subset of the set of memory devices500.

The third memory device506includes a first region530is that statically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aand the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does not change between the available interleaving patterns used in different modes. The third memory device506includes a second region532is dynamically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aor the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does change between the available interleaving patterns used in different modes. For example, the modes (e.g., a high-performance mode and a power conservation mode) that control how the second region532is addressed may be changed responsive to an indication of a use case change for device (e.g., the image capture device100or the image capture device200) that includes the set of memory devices500.

Similarly, the fourth memory device508includes a first region540that is statically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aand the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does not change between the available interleaving patterns used in different modes. The fourth memory device508includes a second region542is dynamically configured to be accessed using virtual addresses that are mapped to physical addresses using an interleaving pattern (e.g., the interleaving pattern600ofFIG.6Aor the interleaving pattern650ofFIG.6B) in a range of virtual addresses that does change between the available interleaving patterns used in different modes. For example, the modes (e.g., a high-performance mode and a power conservation mode) that control how the second region542is addressed may be changed responsive to an indication of a use case change for device (e.g., the image capture device100or the image capture device200) that includes the set of memory devices500. The third memory device506and the fourth memory device508may constitute a second subset of the set of memory devices500.

In this example, the first region510of the first memory device502and the first region520of the second memory device504may be statically configured to correspond to virtual addresses in a first range of virtual address that are interleaved between the first memory device502and the second memory device504. For example, consecutive 512-byte pages of memory in the first range of virtual addresses may alternate between being mapped to the first memory device502and the second memory device504. Interleaving between only two of the four available memory devices in the set of memory devices500may limit the memory bandwidth that can be achieved when accessing data in the first range of virtual addresses. However, statically configuring these regions of the first subset of the set of memory devices500may enable instructions for software stored in this first range of virtual addresses to continue executing while the mode of the processing apparatus is dynamically changed without forcing a reboot.

In this example, the second region512of the first memory device502, the second region522of the second memory device504, second region532of the third memory device506, and the second region542of the fourth memory device508may be dynamically configured to correspond to different virtual addresses in different modes. In a first mode (e.g., a high-performance mode) these regions may be accessed using a range of virtual addresses that are interleaved between all the memory devices (502,504,506, and508) of the set of memory devices500. For example, consecutive 512-byte pages of memory in this range of virtual addresses may cycle between being mapped to the first memory device502, the second memory device504, the third memory device506, and the fourth memory device508. For example, the virtual addresses in this range may be used by software to store heap data (e.g., high-heap data). Using all memory devices of the set of memory devices500in an interleaving pattern for addresses in this range of virtual addresses may increase the memory bandwidth achievable for accessing data (e.g., temporary image data being processed during image capture) stored in these regions. In a second mode (e.g., a power conservation mode) the second region512of the first memory device502and the second region522of the second memory device504may be accessed using a second range of virtual addresses that are interleaved between the first memory device502and the second memory device504. For example, consecutive 512-byte pages of memory in the second range of virtual addresses may alternate between being mapped to the first memory device502and the second memory device504. For example, the virtual addresses in this range may be used by software to store heap data. The second mode may enable the powering down of the third memory device506and the fourth memory device508to conserve power at the expense of reducing the memory bandwidth and capacity available to software for accessing heap data. In some implementations, the boundary between the first region (510,520,530, and/or540) that is statically configured and the second region (512,522,532, and/or542) that is statically configured for any of the memory devices in the set of memory devices500may be adjusted at boot time.

FIG.6Ais a memory map of an example of an interleaving pattern600used to translate virtual addresses into physical addresses in multiple memory devices in a performance mode that uses all available memory devices to achieve high memory capacity and bandwidth. The memory map has two columns arranged from bottom to top in virtual address order corresponding to different ranges of virtual addresses used by a processing apparatus. The left column describes features of the address translation mapping in a range of virtual addresses by identifying which memory device are included in an interleaving pattern applied in that range of addresses. The right column provides examples of data types that software may be configured to store in ranges of virtual addresses to effectively utilize the available memory in the performance mode.

The interleaving pattern600includes a first range602of virtual addresses that is interleaved using only two out of four available memory devices (502and504) in the set of memory devices500. The interleaving pattern600includes a second range604of virtual addresses that is interleaved using the other two out of four available memory devices (506and508) in the set of memory devices500. The interleaving pattern600includes a third range606of virtual addresses that is interleaved using all four available memory devices (502,504,506, and508) in the set of memory devices500. Using all available memory devices for interleaving in the third range606may enable high-bandwidth memory access to data stored in the third range606of virtual addresses. Using all available memory devices in the performance mode also increases the memory capacity available in the system. For example, where each memory device is configured to 1 gigabyte of data, the processing apparatus may make 4 gigabytes of memory available when running in the performance mode.

Software may be configured to store instructions, including instructions of operating system software for the processing apparatus in the first range602of virtual addresses. In this example, software is configured to store real-time operating system (RTOS) instructions610in the first range602; store Linux operating system instructions612in the first range602; store miscellaneous data and instructions614in the first range602; store low heap data616in the first range602and in the second range604; and store high heap data618in the third range606of virtual addresses. By storing the high heap data (e.g., including temporary image data being processed during image capture) in the third range, this data may be accessed with high memory bandwidth for high performance with low processing delays. For example, this high bandwidth memory access may enable the capture of high-resolution video in an image capture device (e.g., the image capture device100or the image capture device200). For example, for high performance, pixel buffers may be stored in the high heap618. For example, algorithms and libraries may be stored in the low heap616.

FIG.6Bis a memory map of an example of an interleaving pattern650used to translate virtual addresses into physical addresses in multiple memory devices in a low-power mode that uses a subset of available memory devices to conserve power consumption. The memory map has two columns arranged from bottom to top in virtual address order corresponding to different ranges of virtual addresses used by a processing apparatus. The left column describes features of the address translation mapping in a range of virtual addresses by identifying which memory device are included in an interleaving pattern applied in that range of addresses. The right column provides examples of data types that software may be configured to store in ranges of virtual addresses to effectively utilize the available memory in the performance mode.

The interleaving pattern650includes a first range652of virtual addresses that is interleaved using only two out of four available memory devices (502and504) in the set of memory devices500. The interleaving pattern650includes a second range654of virtual addresses that is interleaved using the other two out of four available memory devices (506and508) in the set of memory devices500. The interleaving pattern650includes a third range656of virtual addresses that is interleaved using only two out of four available memory devices (502and504) in the set of memory devices500. The interleaving pattern650includes a fourth range658of virtual addresses that is interleaved using the other two out of four available memory devices (506and508) in the set of memory devices500. In this power conservation mode, two out of four available memory devices (506and508) in the set of memory devices500may be powered down (e.g., turned off or disabled) to conserve power consumption. Using only two of the four available memory devices in the power conservation mode may decrease the memory capacity available in the system. For example, where each memory device is configured to 1 gigabyte of data, the processing apparatus may make only 2 gigabytes of memory available when running in the power conservation mode. Using only two of the four available memory devices in the power conservation mode may also reduce the maximum achievable memory bandwidth.

Software may be configured to store instructions, including instructions of operating system software for the processing apparatus in the first range652of virtual addresses. The first range652is the same as the first range602inFIG.6Aand the interleaving pattern650in the first range652matches the interleaving pattern600in the first range602. Having the two interleaving patterns (600and650) for these two modes of operation match in the first range (602/652) of virtual addresses may enable instructions for software to continue to be executed while a switch between the performance mode and the power conservation mode is occurring. In this manner, a reboot of the processing apparatus may be avoided when switching between the two modes.

In this example, software is configured to store real-time operating system (RTOS) instructions660in the first range652; store Linux operating system instructions662in the first range652; store miscellaneous data and instructions664in the first range652; store low heap data666in the first range652; and store high heap data668in the third range656of virtual addresses. Software may be configured to store nothing in the second range654and the fourth range658when running in the power conservation mode so that the third memory device506and the fourth memory device508may be powered down to reduce power consumption. For example, the processing apparatus may be configured to return a memory fault670(e.g., return a bus error) in response to any errant request from software to access addresses in the second range654and the fourth range658when running in the power conservation mode.

A transition from performance mode to power conservation mode or vice versa may be transparent from a firmware point of view. In some implementations, a transition from performance mode to power conservation mode or vice versa only affects heap sizes. In some implementations, a transition from performance mode to power conservation mode or vice versa occurs in response to a camera mode switch (e.g., a change in the video capture resolution). For example, PHY retraining, when turning on one or more memory devices (e.g., a DDR), may be fast (e.g., <100 ms). In some implementations, cold boot will run in performance mode in order to perform initial training of all memory devices in the set of memory devices500. In some implementations, switch retraining only consists of temperature compensation.

FIG.7is a flowchart of an example of a technique700for dynamically switching between modes with different usage of a set of available memory devices. The technique700includes executing702instructions of operating system software for a processing apparatus stored in one or more memory devices in a set of memory devices using a first interleaving pattern to map virtual addresses to physical addresses; invoking704a change from a first mode to a second mode for the processing apparatus while continuing to execute the operating system software, the processing apparatus using the first interleaving pattern for virtual address translation when in the first mode and using a second interleaving pattern for virtual address translation when in the second mode; executing706instructions of the operating system software using the second interleaving pattern, the first interleaving pattern using all memory devices in the set of memory devices and the second interleaving pattern uses less than all of the memory devices in the set of memory devices; and powering down708a subset of the memory devices in the set of memory devices when in the second mode. For example, the technique700may be implemented using the image capture device100ofFIGS.1A-B. For example, the technique700may be implemented using the image capture device200ofFIGS.2A-C. For example, the technique700may be implemented using the image capture device300ofFIG.3. For example, the technique700may be implemented using the processing apparatus400ofFIG.4.

The technique700includes executing702instructions of operating system software for a processing apparatus stored in one or more memory devices in a set of memory devices (e.g., the set of memory devices410) using a first interleaving pattern (e.g., the interleaving pattern600ofFIG.6A) to map virtual addresses to physical addresses. For example, the operating system software may be a real-time operating system (RTOS) software. For example, the operating system software may be a version of Linux. For example, a memory management circuitry (e.g., the memory management circuitry430) may be used to implement the address translation according to the first interleaving pattern.

The technique700includes invoking704a change from a first mode (e.g., a high-performance mode) to a second mode (e.g., a power conservation mode) for the processing apparatus while continuing to execute the operating system software. The processing apparatus uses the first interleaving pattern for virtual address translation when in the first mode and uses a second interleaving pattern for virtual address translation when in the second mode. In some implementations, the change from the first mode to the second mode is completed without rebooting the processing apparatus. The address mapping for virtual addresses associated with the instructions of the operating system software may be the same for the first interleaving pattern and the second interleaving pattern, while the first interleaving pattern and the second interleaving pattern may differ for other ranges of virtual address to accommodate different use cases and power budgets. For example, heap data may be stored in a first range of virtual addresses when the processing apparatus is in the first mode and heap data is stored in a second range of virtual addresses that is smaller than the first range of virtual addresses when the processing apparatus is in the second mode. For example, the change between the first mode and the second mode may be invoked704in response to a use case indication received for a device (e.g., the image capture device100) including the set of memory devices. Some use cases may require more memory capacity and/or memory bandwidth than other use cases. Dynamic switching between the first mode and the second mode may enable optimization for power consumption by enable the selective powering down of memory devices in the set of memory devices without incurring the delays associated with a reboot of the entire device.

The technique700includes executing706instructions of the operating system software using the second interleaving pattern. The first interleaving pattern uses all memory devices in the set of memory devices and the second interleaving pattern uses less than all of the memory devices in the set of memory devices. Using less memory for heap data in the second mode may enable a subset of the memory devices to be powered down708to conserve power consumption. Using less memory devices in the second mode may also reduce the maximum memory bandwidth (e.g., average speed of access) that can be achieved with memory interleaving strategies. For example, heap data may be stored in all memory devices in the set of memory devices using the first interleaving pattern when the processing apparatus is in the first mode and heap data is stored in less than all memory devices in the set of memory devices using the second interleaving pattern when the processing apparatus is in the second mode.

The technique700includes powering down708a subset of the memory devices in the set of memory devices when in the second mode. For example, powering down708a subset of the memory devices may include changing a voltage on an enable conductor of the memory devices in the subset. For example, powering down708a subset of the memory devices may include gating a clock signal into the memory devices in the subset.

FIG.8is a flowchart of an example of a technique800for dynamically running software in a low power mode by using less memory devices to store heap data by changing an interleaving pattern. The technique800includes storing802instructions of operating system software in a first range of virtual addresses; dynamically changing804a mode of a memory management circuitry while executing the instructions in the first range of virtual addresses to select between a first interleaving pattern and a second interleaving pattern implemented by the memory management circuitry for translating virtual addresses to physical addresses in a set of memory devices; storing806heap data in a second range of virtual addresses when using the first interleaving pattern to utilize all memory devices in the set of memory devices for storing heap data; powering down808a memory device in the set of memory devices when using the second interleaving pattern; and storing810heap data in a third range of virtual addresses when using the second interleaving pattern, where the third range of virtual addresses is smaller than the second range of virtual addresses. For example, the technique800may be implemented using the image capture device100ofFIGS.1A-B. For example, the technique800may be implemented using the image capture device200ofFIGS.2A-C. For example, the technique800may be implemented using the image capture device300ofFIG.3. For example, the technique800may be implemented using the processing apparatus400ofFIG.4.

The technique800includes storing802instructions of operating system software in a first range of virtual addresses (e.g., the first range602). For example, the first range of virtual addresses may be reserved for program instructions. In some implementations, a boot routine in non-volatile memory of a processing apparatus may cause the transfer the instructions of operating system software from non-volatile memory to the set of memory devices at locations statically mapped to the first range of virtual addresses. The instructions of operating system software may then be fetched from the set of memory devices using virtual addresses in the first range to execute the operating system software and run a device including the processing apparatus (e.g., the image capture device100).

The technique800includes dynamically changing804a mode of a memory management circuitry (e.g., the memory management circuitry430) while executing the instructions in the first range of virtual addresses to select between a first interleaving pattern (e.g., the interleaving pattern600ofFIG.6A) and a second interleaving pattern (e.g., the interleaving pattern650ofFIG.6B) implemented by the memory management circuitry for translating virtual addresses to physical addresses in a set of memory devices. The first interleaving pattern and the second interleaving pattern are the same within the first range of virtual addresses (e.g., the first range602) and there are differences between the first interleaving pattern and the second interleaving pattern outside of the first range of virtual addresses. The preservation of the mapping of addresses in the first range of virtual addresses between the first interleaving pattern and the second interleaving pattern may enable the instructions in the first range to continue to be executed without rebooting the processing apparatus. The differences between the first interleaving pattern and the second interleaving pattern outside of the first range may enable the processing apparatus to dynamically adapt memory usage in the processing apparatus to suit different use cases while conserving power or increasing memory capacity and memory bandwidth. For example, dynamically changing804a mode of a memory management circuitry may be triggered in response to a use case indication received for a device (e.g., the image capture device100) including the set of memory devices. Some use cases may require more memory capacity and/or memory bandwidth than other use cases.

The technique800includes storing806heap data in a second range of virtual addresses (e.g., the third range606) when using the first interleaving pattern to utilize all memory devices in the set of memory devices for storing heap data. By storing the heap data (e.g., including temporary image data being processed during image capture) in the second range, this data may be accessed with high memory bandwidth for high performance with low processing delays. For example, this high bandwidth memory access may enable the capture of high-resolution video in an image capture device (e.g., the image capture device100or the image capture device200).

The technique800includes powering down808a memory device in the set of memory devices when using the second interleaving pattern. For example, powering down808the memory device may include changing a voltage on an enable conductor of the memory device. For example, powering down808the memory device may include gating a clock signal into the memory device.

The technique800includes storing810heap data in a third range of virtual addresses (e.g., the third range656) when using the second interleaving pattern. The third range of virtual addresses is smaller than the second range of virtual addresses. In some implementations, the third range of virtual addresses is a subset of the second range of virtual addresses. For example, the third range of virtual addresses, despite limitations on memory capacity and bandwidth, may be sufficient to support operations of the device in a power conservation mode.

While the disclosure has been described in connection with certain embodiments, it is to be understood that the disclosure is not to be limited to the disclosed embodiments but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures as is permitted under the law.