Abstract:
Apparatus, methods, and systems are disclosed to manage memory in an embedded system. The system registers video applications and video sources with a memory manager. The memory manager in turn provides memory to the video applications and video sources. The system has an input to receive an output from at least one video source. The memory manager receives a frame from the video source and transfers the frame to memory. Once the frame is in memory the video application may work with the frame. All of these operations are conducted with the memory manager actively managing and allocating the memory resources.

Description:
BACKGROUND 
     Video systems utilize video applications, which may be described as components in software that manipulate video, particularly video acquired from a camera. Video applications require large amounts of memory. Video applications manipulate one or more frames of video, often in a way that requires the entire frame or frames to be in memory all at once. Examples include: sending video out via TCP, sending video out via UDP, rendering video on a local display, transcoding to a different video type such as a series of raw images to JPEGs, motion detection, etc. The individual frames may be quite large, so running multiple video applications simultaneously, each holding multiple video frames, results in very high memory usage. 
     Video applications work with the embedded system to manipulate or utilize information provided from video sources such as cameras. Video applications are best designed to be encapsulated from the details of the video source. This allows for the optimal reusability and extensibility of the video application and the video source software (i.e. the camera driver). 
     The embedded systems often have a small amount of memory especially when compared to modern PCs. It is a challenge to provide an optimal amount of PC-like features in an embedded device with much less memory than a PC has. For embedded systems there is a wide spectrum of available horsepower and costs for processor speed and the size of available memory. Generally, as cost decreases, horsepower and the available memory decreases. As available memory decreases, supporting video becomes more of a challenge because of the memory requirements of video. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a system diagram of an embodiment of the invention. 
         FIG. 2  is a flow chart of a method for managing buffers for inputs from video sources according to an example embodiment. 
         FIG. 3  is a flow chart for the video application&#39;s interaction with the memory manager according to an example embodiment. 
         FIG. 4  is a flowchart for a method for configuring a memory manager according to an example embodiment. 
         FIG. 5  is a flowchart for a method for utilizing a software component to adding new memory manager qualities of service dynamically according to an example embodiment. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is a system diagram of an example embodiment. The system  100  incorporates an embedded system  110  and multiple camera inputs. A plurality of cameras may be connected directly to the embedded system  110  such as by a USB (Universal Serial Bus) connector. The cameras may also be connected wirelessly to the embedded system  110 . Cameras  120 ,  121 ,  122 ,  124 ,  126 , and  128  may be connected directly to the embedded system  110 . These connections  101 ,  103 ,  105 ,  107 , and  109  may be USB connections, Firewire connections, or any compatible connection method. A wireless connection may include an antenna  123  connected to the embedded system  110  via a transceiver  127 . A camera  121  may be wirelessly connected to the embedded system  110  by antenna  125 , by communicating with the embedded system  110  through antenna  123 , or through a router  170 , which may be wirelessly enabled and have an antenna  178 . 
     Embedded system  110  is enabled to accept from cameras  120 ,  121 ,  122 ,  124 ,  126 , and  128  video in a format such as JPEG. JPEG is a commonly used method of compression for photographic images. The name JPEG stands for Joint Photographic Experts Group, the name of the committee that created the standard. While the specification shall discuss the operation utilizing the JPEG format, other formats of video capture may be utilized with the embodiments of the invention. 
     Embedded system  110  may be connected to peripherals, a network such as an Ethernet network, or the internet  177  via the router  170  and/or a modem  175 . Modem  175  may be connected to a server  180  through the internet  177 . The embedded system  110  may be connected to a personal computer  182  via an Ethernet connection  150 . Personal computer  182  may also be connected to a printer  186 . Embedded system  110  may also be connected to a monitor  184 . Monitor  184  may be connected as shown directly to the embedded system  110  through a USB connection  153  or through an Ethernet connection (not shown) via router  170 . A personal computer  188  may also be connected directly to embedded system  110  via a USB connection  155 . Personal computer  188  may also be connected to a printer  189 . 
     The embedded system  110  may communicate with peripherals via wireless connections. For example, embedded system  110  may communicate to a personal computer  134  having an antenna  136  via antenna  123  connected to transceiver  127  or antenna  178  through router  170 . Additionally, a PDA  130  (personal digital assistant) having an antenna  132  may be connected wirelessly to embedded system  110  via antenna  123  or antenna  178 . The wireless connections may utilize a Wi-Fi, infrared, or other wireless connection means. Wi-Fi refers to a family of related specifications (the IEEE 802.11 group (Institute of Electrical and Electronics Engineers)), which specify methods and techniques of wireless local area network operation. It is understood that other wireless connection methods may be utilized, provided the wireless connection method provides at least one-way communication either to or from the embedded system  110  to the wireless device. 
     Embedded system  110  may incorporate memory  115  (such as RAM, random access memory) to receive the direct line inputs from one or more cameras  120 ,  121 ,  122 ,  124 ,  126 , or  128 . Embedded system  110  may also incorporate a processor  119  and operating software  111 . The operating software  111  may be stored in non-volatile memory  112  and may be stored either in the non-volatile memory  112  or in the memory  115  for execution. Non-volatile memory  112  may be a hard drive, flash memory, or other non-volatile memory. Video Management Quality of Service  114  software (VMQOS) may also be stored in non-volatile memory  112 . VMOQS may be moved to volatile memory for execution or operated from non-volatile memory  112 . VMQOS  114  software may specify that a memory reserve  117  be allocated in RAM  115  to receive video inputs from cameras  120 ,  121 ,  122 ,  124 ,  126 , and/or  128  or for software applications that may be stored in non-volatile memory  112 . The size of the specified memory reserve  117  may be set by VMQOS  114 , VMQOS may receive inputs from a user through one of the peripheral devices, or by an API from a camera or other device. An application programming interface (API) is a software interface that a computer application, operating system, or library provides to support requests for services to be made of it by a computer program. The memory reserve  117  size may not be a fixed size and may vary based upon the operation and requirements of the embedded system  110 . In addition to specifying the volatile memory  115  allocations for incoming video, VMQOS may also specify the amount of memory available for embedded software in the embedded system  110 . 
       FIG. 2  is a flow chart of a method for managing buffers for inputs from video sources according to an example embodiment. Method  200  may include activity  210  to start the video source. The video source may be a camera such as camera  120 ,  121 ,  122 ,  123 ,  124 ,  126 , or  128  of  FIG. 1 . Activity  220  may be to have the video source register with the memory manager. As stated earlier, the video source may register with an API to define the memory requirements of the video source. Activity  225  may be to determine if the video source is acquiring video. If the video source is not acquiring data, activity  227  may unregister the video source with the memory manager and activity  228  stops activity by the memory manager for that video source. If the video source is acquiring video activity  230  is to have the video source request buffer or memory from the memory manager. Activity  230  the video source informs the memory manager how much buffer the video source would like to have. The memory manager operates the VMQOS  114  software of  FIG. 1  to determine how much memory should be allocated to the video source. Activity  235  may be to have the memory manager return a buffer ID (identification) to the video source. Activity  240  may be for the video source to provide data to memory manager and request that the memory manager write the data to the identified buffer. Activity  245  may be for the memory manager to write the data to the buffer identified, discard the data or pace. Pace means the memory manager determines that memory may not be available at that time and the memory manager will delay the video source until the memory manager determines there is space available. Activity  250  may be to return control of the function to the video source. Activity  255  may be for the video source to determine if the end of the frame has been reached. If the end of the end of the frame has not been reached the process returns to activity  240 . If the end of frame has been reached, activity  260  may be to have the video source close the buffer ID. Activity  225  is then initiated to determine if the video source continues to acquire. 
       FIG. 3  is a flow chart for the video application&#39;s interaction with the memory manager according to an example embodiment. The method  300  operates to allocate memory for video applications. Activity  310  may be to start the video application. Activity  315  may include having the video application determine what source the application will consume. Activity  325  may be to have the video application decide if one or more buffers are needed and if the buffers are to be “long lived”. A “long lived” buffer is one that may need to be available for an extended period of time as determined by the video application. If the video frame is needed for a longer time than the frame period, the video application will need a long lived buffer. The frame period is the inverse of the frame rate and is the amount of elapsed time between frames. Activity  330  may be to have the video application register with the memory manager. The video application may register interest in a video source as defined in activity  315 , and may request that that when a new frame is available that the video application be notified via a frame callback. A frame callback is a message originating from the memory manager to the video application indicating a new video frame is available. Activity  335  may be for the message manager to determine if it will make a frame available. If a frame is not made available activity  337  may be to wait until a new frame call back is received from the memory manager. When a frame is available the memory manager to provide a message to the video application (a new frame callback message) indicating a new frame is available. Activity  340  may be for the memory manager to provide additional information to the video application such as the size of the frame, location of the frame and whether it is asynchronous or synchronous. An asynchronous versus synchronous determination indicates to the video application whether or not the memory manager will wait on the video application before continuing its work with that video frame. Synchronous indicates the memory manager will wait for the video application to complete its work and asynchronous indicates that the memory manager will continue to process that frame independent of what the video application does. In addition the memory manager may request immediate return which is a way for the memory manager to indicate to the video application that the video application must complete work with that frame as quickly as possible. This indicator may be used by the video application as a signal to skip this frame if the video application knows it cannot be processed quickly or the video application may take whatever other action it determines in order to return a complete message to the memory manager. Activity  345  may include having the video application process the frame based on the memory manager message of activity  340 . Activity  350  may be to have the video application provide a message to the memory manager that the frame buffer processing is complete. When activity  350  is completed activity  355  may be to determine if the video application is done. If the application is not done, activity  335  may repeated for the next frame. If the application is done, activity  360  will be for the video application to unregister with the memory manager. 
       FIG. 4  is a flowchart for a method for configuring a memory manager according to an embodiment of the invention. Method  400  may be to begin with activity  410  initiated by the client accessing the memory manager configuration interface. This may be a user interface available, for example, on personal computer  182  or  188  of  FIG. 1 . Activity  420  may include having the memory manager provide information to the client. This information may include the configurable parameters, information on all sources that are currently available and have been historically available. The information provided may also include information on all video applications currently and historically available for use. Finally, the user interface may include configurable quality of service options. The quality of service options may determine for example the resolution of video frames, prioritize video sources and video applications. For example, if a door is opened it may make the video source monitoring that door the priority based on preferences set by the user. Activity  430  may be to have the user configure the memory manager based on choices that were made available to the user. 
       FIG. 5  is a flowchart for a method for utilizing a software component to adding new memory manager qualities of service dynamically according to an example embodiment. A software component may be written that conforms to the pluggable quality of service API of the memory manager. The new software component becomes selectable by a user (or any agent capable of configuring the system). The API allows the memory manager to delegate memory usage decisions to the new software component when it is activated. The activated software component will be notified by the memory manager about video events. Based on the video events, the activated software component will configure the memory manager, video sources and video applications to affect their behavior as programmed in the software component. 
     Method  500  describes the operation of a new pluggable quality of service software component. Activity  510  may be to create a new software component by implementing the pluggable quality of service API of the memory manager and by exposing configuration information. Activity  520  may be to register the new software component with the memory manager. By registering the memory manager is made aware of the new software component. Activity  530  may be for the new software component to wait for the memory manager to either activate the software component or to un-register the software component. Activity  540  may be to determine if software component has been activated or un-registered. If it has been activated, activity  545  may be to wait until either a video event is provided by the memory manager or a deactivation instruction is provided by the memory manager. Activity  550  may determine if the software component has received a video event from the memory manager. If a video event has been received, activity  560  may be for the software component to perform an action based on the video event. The action performed in activity  560  may be that the software component modifies the configuration of the memory manager, any video sources, and/or video applications. The video event, may be for example that a new frame is available from a video source, or a video application has requested a long lived buffer, or a video application has freed a long lived buffer. 
     If activity  550  determines that a video event has not been provided in activity  545 , but a deactivate command has been received, activity  570  may be to deactivate the software component. Once the software component is deactivated activity  530  is repeated. 
     If activity  540  determines the software component was to be unregistered, activity  580  may be to un-register the software component and perform any cleanup of resources the software component needs to, as determined by the software component activity. If the system or user determines that the software component needs to be used again, it may be registered in activity  520 . 
     The Abstract of the Disclosure is provided to comply with 37 C.F.R. §1.72(b) requiring an abstract that will allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. The above description and figures illustrate embodiments of the invention to enable those skilled in the art to practice the embodiments of the invention. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.