Patent ID: 12206947

DETAILED DESCRIPTION

Input Module

Embodiments of the present system may be implemented and/or deployed on one or several computers, monitoring workstations, and/or homologous processors. With reference toFIG.1, media streams are received by a processing system from one or more external sources. The specific types of sequences of data elements that make up a media stream accepted by one or more embodiments of the present invention may be restricted to a specific set of video data having a specific encoding or video compression format, which as an exemplary enumeration, but without limitation, may include, H.264, H.265, MPEG-4, or MJPEG.

In other embodiments, the types and formats of data streams accepted may be far less restricted. Accordingly, sources099for the aforementioned media streams may unlimitedly include any one or more of an image or video capture device such as a camera, a local hard drive, media server, or remote network location. Additional sources for input media streams may include sensors capable of receiving, measuring, or converting information or data—whether simultaneously or non-concurrently—that is contextually relevant to the aforementioned media stream sources. Such additional input media sources may include, without limitation, sound, GPS coordinates, human, biometric, object identification, validation, or other telemetric data.

In other embodiments, metadata resulting from more complex event processing may form part or all of a media stream. In various embodiments, the acceptability of an input media stream101,102,103follows said stream's intelligibility to the system; such as may be enabled, in several embodiments, through the presence of one or more appropriate media codecs or other software or hardware capability, to decode or otherwise render intelligible one or more streams stored in one or more different formats.

In at least one embodiment of the present invention, input media streams101,102,103, each whether of identical or differing types, and in accordance with the capacity of the system, are received099by the input module100for handling by said embodiment. In a further series of embodiments, said input module100may passively receive or actively fetch input media streams from any source099external to the system, or admit a mixture of these stream inputting means. In a still further series of embodiments, streams received by the input module100may be duplicates or dedicated local copies of those received from external sources099for the exclusive use of embodiments of the present invention. Such dedicated copies may be desired for the added efficiency and convenience they offer to the deployment or execution of an embodiment of the present invention, such as to minimize latency at the moment of processing, or to help ensure the integrity of one or more input streams101,102,103. In a still further series of embodiments, streams received by the input module100may be implicit or explicit references to said streams101,102,103. Said access of streams101,102,103by reference in lieu of an implicit duplication streams is ideally governed by an appropriate semaphore or equivalent mechanism to support mutual exclusion of said stream use by multiple systems for handling by one or more embodiments of the present invention.

In another series of embodiments of the present invention, the organization and layout of the input module100may vary. As with various modules further described herein, input module100need not be understood to be necessarily contiguous, either in a physical or conceptual sense; for example, portions of the module may be implemented in just one or alternatively across multiple locations and interconnected through a network. In at least one embodiment, for example, one or more additional input modules100′ may coexist and correspondingly provide input media streams101′,102′,103′ to the system. The purpose of such additional input modules100′ may reflect any number of abstractions or criteria, including without limitation, additional physical locations of streams, distinct stream fetching and/or access policies, or specific types or classes of image or other data contained. In certain deployments for example, streams contained within one or more input modules can be stored using an AVPacketstruct of the FFmpeg library, known in the art.

In addition to the one or more media sources099described previously, input module100and any one or more complements101′ communicate information, when so inspected and detected150, about the payload of every stream101,102,103entering the input module100for handling by an embodiment of the present invention. Extraction of such information is key to allowing embodiments of the present invention to identify each of the input media streams101,102,103received from external sources099to prepare embodiments of the present invention for the eventual processing tasks to be performed on them, as will be further described herein. Details surrounding the necessary identification and subsequent communication of such information150regarding said streams101,102,103will be discussed presently.

Stream Detection Engine

Knowledge about the specific nature of input media streams101,102,103received by the input module100is vital to the function of embodiments of the present invention. The behavior of modules further discussed herein and indeed progress and execution under different operational scenarios and circumstances depend significantly upon the types of data received by said embodiments. Accordingly, embodiments of the present invention may obtain such knowledge via any one or several means. A stream detection engine200, dedicated to the positive identification of incoming streams101,102,103, provides an expedient modular abstraction obtain such knowledge.

For example, a finite number of stream input types known to some embodiments of the invention may be recognized in said embodiments through a parsing of corresponding streams' packet headers. In further embodiments, a detection algorithm may be present to arrive at a positive identification of streams through payload analysis of said streams (FIGS.6,7). In further embodiments, a combination of the foregoing (such as through executing various strategies in parallel or in a cascaded manner), in addition to any one or more supplementary identification schemes not discussed herein may be envisioned for purposes of ascertaining the type of stream(s) received. In still further embodiments, additional information consisting of specific details about said stream(s) pertinent to the processing, handling, and specific deployment objectives of said embodiments may likewise be sought, for example through one or more deep packet inspection and analysis schemes. As a non-limiting example, additional identification information sought regarding stream inputs101,102,103may include, in the case where said streams are video streams for instance, said stream inputs' framerate and resolution.

Of course, the broad input stream reckoning strategies presented above are not intended to be an exhaustive enumeration of all possible approaches that may be envisioned for all embodiments, but are merely a suggestion of certain strategies that may be envisioned for deployment. In still further embodiments, it is possible that a type or even a specific aspect of one or more input streams may not be positively or definitively identified at all; in such cases, said streams may be identified as such and an error handling approach suited to the deployment objectives of said embodiments may be executed. These may include, in some non-limiting examples, notifying or otherwise prompting a human operator for specific action, and attempting to proceed nonetheless.

In some embodiments, the stream detection engine200may be configured to have limited access to the input module100for purposes of detecting the type and other attributes of streams101,102,103received by said module100. The stream detection engine200may likewise be configured, in certain embodiments, to inspect150the input module100with one or more reckoning strategies such as discussed herein at a specified interval. In other embodiments, the engine200may be configured to dynamically or asynchronously detect the addition of a stream101,102,103to the input module100and perform detection and query operations1500nthe new stream added. Likewise, further embodiments where such stream detection and payload query operations150are carried out dynamically or asynchronously may be further configured to provide such detection abilities on an ongoing basis upon said media streams101,102,103for the entire time that said streams are available within the input module100, rather than merely once. Multiple inquiries150of the streams101,102,103over the interval during which they are present within the input module100allow embodiments of the present invention to detect changes within the nature of a specific input media stream (including, but not limited to a change in a video stream's frame rate, resolution, color space), and allow other modules further discussed herein, to respond to such changes by modifying the behavior of the system as appropriate. In an embodiment of the present invention, an inspection and detection operation150may require or otherwise include a granting, to the stream detection engine200, of proper mutual exclusion or access permissions through a synchronization mechanism of any one or more input media streams101,102,103prior to actual detection or parsing of any part of said streams.

In deployments, implementations of the stream detection engine can, for example, make use of the FFmpeg library's core functions/structures and utility functions to identify the nature of a stream available to the input module100.

Following identification of a stream, the engine200generates a stream payload analysis result250. The particular encoding standard and format of said analysis result250may vary in accordance with the deployment needs and scenarios of specific embodiments of the present invention. In some embodiments, a single result250at a time may be returned; in other embodiments, a series of such results collected over a given time interval may be compiled or otherwise merged and retained in accordance with a given condition or policy. In a related embodiment, said results retention (including, without limitation, number of individual records and the amount of time each is kept) may be configured as a parameter of the system, as discussed further herein. In a further embodiment, the analysis results250may likewise be requested or otherwise communicated in accordance with similar considerations and needs of said embodiment. As a non-exhaustive enumeration, the engine200may be configured in such a way as to communicate or share such results250once the detection operation for an individual stream101has completed, or once the nature and attributes pertinent to the respective embodiment of the present invention have been identified for a number of streams101,102,103.

In another series of embodiments, particularly those in which rapid identification of some aspect(s) of input stream types is of particular or time-sensitive importance (such as in a mission-critical deployment scenario in which a large number of input streams is received by an embodiment of the invention in a short period of time), the specific record of a payload analysis result250may configured for various levels of granularity and to query and deliver certain aspects of the analysis result250deemed of greater importance before others. In such embodiments, for example, it might be necessary or desirable for an embodiment of the present invention to ascertain the video format, frame rate, and resolution of a particular stream input as a coarse means by which to arrive at a heuristic to assess and anticipate the processing load associated with said stream input101. In said embodiments, returning such information prior to performing any deep packet inspection of said input's101bit stream to determine information deemed of secondary or lesser importance, is useful. A subsequent result250with greater or full granularity may be requested and provided subsequently and/or at a later time when less input module100traffic is received by an embodiment of the invention. In a further embodiment, a priority policy may be specified or configured such that additional or full information about one particular (or particular category) of stream may be provided in a result250before comparable information is compiled and/or ascertained by the engine200for another (category of) input stream.

In some embodiments, the stream detection engine200can be configured with information about the stream types supported by the system. In such embodiments, results record250may accordingly contain information specifying whether an incoming stream is of a type that is partially recognized or even entirely unsupported by the system, in which case corrective action (including but not limited to execution of the aforementioned error handling procedures) may be undertaken.

Operator Input Module

It will be appreciated that for various embodiments of the present invention, the modules discussed herein may accept configuration settings of various kinds to allow said embodiments to alternatively alter, adjust, vary, or simply set their behavior in accordance with the deployment objectives or scenarios of said embodiments. Such configuration settings, which may affect any one or all modules described herein, may be specified, in various embodiments, by one or even several human operators. In further embodiments, access policies or privileges may be specified to variously allow or restrict the specifying of some or all of the configuration settings available to said embodiments by specific users or, in still further embodiments, by categories of users.

Categories of users may include, without limitation, human operators of an embodiment of the present invention for whom a limited interaction profile has been specified. In an embodiment, such limited interaction may include access to or restriction from using any one or more of the modules described herein, whether in whole or in part. In a further embodiment, categories of users may include, to provide a non-exhaustive enumeration, system integrators, deployment professionals, human operator superusers, and system maintenance staff. Indeed, in a still further series of embodiments, the notion of “user” need not be limited to a human operator or other class of human individual with a specific level of access to any whole or part of the system; in such an embodiment, the notion of what constitutes a user may be extended to include an external non-human actor, such as a third-party computer program or related application programming interface providing partial or total access to one or more aspects of embodiments of the present invention. In a still further embodiment of the present invention, examples of human users might include law enforcement and security personnel.

In various scenarios, interactions between embodiments of the present invention and humans and/or external non-human actors may ultimately affect the operation of various modules of embodiments of the present invention. Nonetheless, it is convenient to visualize all such interactions as being received by or otherwise consolidated within a dedicated operator input module300. Consistent with this conceptualization, it is from said operator input module300that configuration information and instructions350to embodiments of the present invention emanate. Such configuration and instructions350may contain key policy or function-specific parameters for the operation of embodiments of the present invention, and are further discussed herein. In particular, aspects of modules described herein as being configurable may for convenience be integrated or featured as (including but not limited to a graphical user interface, menu, submenu, or control panel) encapsulated within the operator input module300(FIG.8). In another embodiment, the operator input module300may receive external input using any appropriate human interface device, such as a mouse, keyboard, or touchpad. In a further embodiment, such input may be received from a system or application external to an embodiment of the present invention, with said reception being implemented via any compatible message passing scheme known in the art.

In certain embodiments of the present invention, the scope of system parameters that may be specified via the operator input module300may be severely restricted or, in further embodiments, be entirely absent. In the latter case, such parameters may be independently specified within other modules described herein. In still further embodiments, said modules may be configured to operate in accordance with a contained set or finite number of combinations. In still further embodiments, such parameter selection may be carried out autonomously by the system's various modules, obviating the need for an operator input module300entirely. In such embodiments, parameters such as the maximum allowable load to place on one or more processing resources, further discussed herein, may be specified. Likewise, it may be expedient in further embodiments to specify, within the operator input module300, additional policy pertaining utilization patterns of resources present to adopt or, in related embodiments, to prefer.

For example, in certain embodiments of the present invention, utilization policies for various processing resources501,502,503, may include, without limitation, instructions350on whether to enable specific hardware acceleration capabilities (when such capabilities are present) as further discussed herein; such instructions350may be specified within the operator input module300. In further embodiments, such instructions350may likewise include a list detailing the sequence of specific processing operations to perform on input streams101,102,103.

In still further embodiments, portions of such instructions350may be received from one or more modules external to an embodiment of the present invention and routed through the operator input module300before being issued and applied to the system. In still further embodiments of the present invention, an optional display management module900, further discussed herein, may alternatively or complementarily receive said portions of said instructions350before routing the latter to an implementation of the operator input module300.

In further embodiments yet, a user can visually assess and specify, within an implementation of the operator input module300, which media stream101,102,103contained within and corresponding to a specific tile within a display900(further discussed herein) has encountered a decoding or other processing malfunction, error, or failure. The user can in implementations be provided with interface controls to manually identify the tile in which a malfunction has been observed. As a result of such identification, corrective or recovery action can be taken. For example, a failure involving a decoding operation for a specific stream101using a specific processing resource501can be recovered from. In such a scenario, a monitoring workstation can, further to the foregoing user identification, implement a mechanism by which a recovery of the stream101and failed processing resource501can be attempted, such as by diverting processing to a different processing resource502. In a still further implementation, the specific processing resource502to which to divert processing can be specified by the user via an interface provided by the operator input module300.

Recovery following a resource crash can thus consist of four broad steps.

The first of these consists of detecting that a recoverable error occurred in a given resource, such as a hardware decoder. Such detection can be provided in the form of software or API-based reporting capabilities of the resource itself, or via an external observation or inquiry made of a resource. The second step consists of substituting a failed or failing resource (such as a GPU-based hardware-implemented decoder) with a more robust and operational resource (such as a CPU-based software-implemented decoder) available within the resource pool500. It will be appreciated that such a substitution should be carefully carried out so as to minimally impair the operation of any other aspect of implementations described herein. It will be appreciated that in certain scenarios, image frames exiting a newly substituted software-implemented processing resource might require additional substitution of subsequent or related processing steps. This might happen, for instance, in cases where processing consists of decoding as well as rendering steps. In such cases, as software frames exit a newly-allocated software-based decoder, the previous hardware-based renderer is likewise substituted with a software-based renderer. In the meantime, the crashed or problematic hardware decoder is quarantined and terminated once processing has been safely assigned away from it. It will be appreciated that even in cases where GPU-based resources (e.g. decoders) are robust with respect to potential packet and/or frame loss, driver errors can prove particularly difficult for security system video monitoring workstations involved in processing and displaying large numbers of streams to recover from.

In the foregoing scenario, the crash recovery mechanism is said to be seamless because the software decoder thus substituted immediately resumes processing duties where the hardware decoder left off. From a user standpoint, such a switchover represents a minimal interruption, as video playback can in many cases be imperceptibly affected. From a deployment- or implementation-centric perspective of monitoring workstations*, it will be appreciated that in the absence of control over stream parameters of input media streams101,102,103in addition to processing resource implementations themselves, crash recovery operations as described herein can prove particularly useful.

Resource Monitoring Engine

It will be appreciated that in certain embodiments of the present invention, it is advantageous to dedicate a module to monitor the state of processing resources501,502,503available500to the system overall. The pool of processing resources available to embodiments of the system500are mentioned in the present section merely for contextual relevance; they500are discussed in further detail and within a more dedicated section herein.

It will be further appreciated that rigorous monitoring of such processing resources500,501,502,503, is appropriate and indeed useful in high performance or mission critical embodiments. In particular, combining such monitoring with a feedback reporting and a corresponding reaction mechanism—in cases where these are needed or desirable, renders the implementation all the more robust and valuable. The foregoing considerations provide a compelling impetus for at least certain embodiments of the present invention to include a resource monitoring engine400. While the various functions of such a monitoring engine400may in certain embodiments be contained within an explicit and dedicated module, it will be further appreciated that such functions might, in other embodiments, be limited or otherwise encapsulated within other modules of the system or in still further embodiments be entirely absent.

In embodiments where it is present, the resource monitoring engine400monitors450the load of the processing resources501,502,503that make up the processing pool500of said embodiments of the present invention. The nature of such monitoring, including but not limited to the frequency with which it occurs and indeed the granularity of the usage analysis to perform, may vary in accordance with the specific objectives and deployment scenario of each specific embodiment of the invention (FIG.9).

In certain embodiments, monitoring450of the resource pool500may likewise follow a custom and non-uniform policy for the entirety of resources501,502,503contained within the resource pool500. For example, one processing resource501may be subject to a load monitoring450policy or even specific load reporting requirements that may differ from those of502. In various embodiments of the present invention, such differences may include, without limitation, the frequency with which each one or more resources is monitored, the type, category, or specific name of the processing operation done by each at the time at a given moment, in addition to any other statistical information (including but not limited to active time, disk usage, or other performance metrics) deemed valuable for the specific scenario and purpose in which said embodiments are deployed.

In a further series of embodiments, the load monitoring information450may be stored within the resource monitoring engine400until said engine400is queried for such information450in a manner analogous to that seen in the stream detection engine200. In a still further series of embodiments, the resource monitoring engine400may be configured to retain such information in accordance with a given criteria set, such as a specific period of time. In such embodiments, the resource monitoring engine400may store such data while the engine400awaits a resource utilization data query640from the system's attribution module600, the latter of which will be further described in a subsequent section herein. Nonetheless, said attribution module600receives the aforementioned stream payload analysis result250, the system configurations and specific operational instructions350, in addition to completing a resource utilization data query640, all in an effort to generate an overall directive to dictate and manage processing to be carried out by the system.

In a related series of embodiments, in the course of monitoring450, the resources501,502,503may be polled for their status or for details regarding their operation, with the result of such polling being communicated back to the resource monitoring engine400. In a further series of embodiments, specific policies on said polling and any related monitoring activity450may be specified as part of the configuration information350, the latter of which is indirectly shared with the resource monitoring engine400by the attribution module600in the course of a resource utilization data query640that the latter600issues to the former400. In a further related series of embodiments, the result of such polling450may be compiled by the engine400into a matrix or matrix-like series of data which matches the resource501,502,503with any number of load and/or performance parameters and with tasks currently occupying the aforementioned resources, with said resulting matrix forming the basis of the resource utilization data query640provided to the attribution module600by the engine400. It will be appreciated that the information populating the aforementioned matrix and delivered to the attribution module600may likewise deliberately constitute a limited or partial (rather than integral) set of data available. Such limitation might, in a manner analogous to that discussed previously for stream detection, be desirable in embodiments or even in specific operational situations in which a specifically defined minimal and/or rapidly collected quantity of information are sufficient for use by the attribution module600.

Likewise, the ability of the resource monitoring engine400to statically or dynamically monitor450the operational health and status of any one or more processing resources501,502,503within the resource pool500can prove particularly useful. In particular, such monitoring450can be advantageously exploited to detect a processing error or a failure in a specific processing resource501,502,503within the resource pool500and contemporaneously allocated to a specific input media stream101,102,103.

In another series of embodiments, the resource monitoring engine400may be entirely absent by design. In such cases, no load monitoring of the resource pool500need be done450or accordingly queried640. Such embodiments may typically be implemented in situations and/or on systems characterized by such a conservative usage policy of resources present in the processing pool500as to obviate the need for such monitoring capability.

Resource Pool

It will be appreciated that embodiments of the present invention require the ability to process data commensurate both with the streams101,102,103received099as inputs100as well as the desired processing operations to perform. Accordingly, it is convenient to contemplate the processing ability of embodiments of the present invention as a resource pool500comprising any of several identical or distinct processing resources501,502,503. Said resources501,502,503may be processors of any kind, and accordingly need not be constrained to any one type or technology. The resource pool500may be made up of any number of processing resources501,502,503, each having any microarchitecture and electrical layout operative to permit the processing tasks required for scenarios in which the various embodiments of the present invention will be deployed. Thus, in various embodiments, said processing resources may be one or more general purpose CPUs, including but not limited, in at least one embodiment, to the one or more CPUs on which the present system executes. The processing pool may likewise comprise, for any one or more embodiments of the present invention, one or more GPUs, ASICs, DSPs, physics processing units, image processors, network processors, audio processors, or any other processing means501′,502′,503′. Indeed, such processing resources may include GPUs for video processing, particularly for use in compute-intensive processing operations and for purposes which include freeing the CPU for other tasks. For example, GPUs running the Maxwell microarchitecture can be envisioned as constituting one of several possible desirable GPU-based hardware processing resources among those found in the resource pool500. These can further be of a type compatible for use with the CUDA application programming interface for general purpose processing.

For their part, GPUs provide two broad types of resources of particular interest to embodiments described herein. One type of such resources is the array of programmable cores, referred to in documentation by NVIDIA as “CUDA cores” and as “shaders” in Direct3D documentation. Such programmable GPU cores correspond to a processing resource having a large amount of cache and a very large number of slow and unsophisticated cores as compared to a CPU, or even an FPGA. On the other hand, GPU cores can advantageously and repetitively execute a simple processing task, such as computing the color of every pixel of the screen, a very large number of times. This contrasts with the advantageous abilities of CPUs, which excel at complex branching logic. A second type of resources procured by GPUs is purpose-built hardware dedicated to decoding a range of specific video compression formats, such as H.264 and HEVC.

Resource vendors, typically but not necessarily the hardware device manufacturers, provide and implement application programing interfaces (APIs) as discussed herein to access the aforementioned processing functionality.

Such vendors typically provide and implement certain APIs for accessing this functionality from software. Accordingly, some APIs are defined and published by the hardware manufacturer (such as CUDA, NVCUVID and NVAPI), while some can be defined by an external entity but implemented by a different manufacturer (such as Microsoft's Direct3D or DXVA).

The Applicant has further determined that decoding functionalities enabled by various general-purpose decoding libraries available for execution on a CPU platform are particularly desirable in this context because of the comparatively robust operation that they provide. The libavcodec encoding/decoding library included within the FFmpeg free software project is one such example. In particular, such decoding libraries, by virtue of their comparatively lengthy development history and breadth, typically provide significant resilient operation in the event of departure of a stream from an established standard. This contrasts with decoding libraries typically available and designed for execution on a GPU platform. Nonetheless, even the FFmpeg project, for instance, provides a subsystem to enable acceleration using hardware devices. The latter makes it possible to use specific devices (including GPUs) to carry out multimedia processing. Furthermore, various acceleration API implementations are available including those enumerated at <https://trac.ffmpeg.org/wiki/HWAccelIntro>. Likewise, when the processing abilities of said processing means other than the central CPU of a workstation are exceeded, a portion of the workstation's own CPU's load may in some embodiments be procured to provide surplus processing capability to meet said demand. Other processing operations may include decompression (whether as part of or separate from a generic understanding of decoding), image scaling, as well as any other processing task.

It will also be appreciated that in embodiments of the present invention, the processing resources present should possess the specific capabilities necessary to perform the processing operations required by the system. For example, whereas typical surveillance cameras provide a framerate in the vicinity of 30 fps, security walls and their often underlying security system video monitoring workstations are required to support cameras outputting a framerate closer to 120 fps, as isolating cases of cheating by way of sleight of hand, in addition to broader security requirements, constitutes a relevant and pertinent preoccupation.

To determine whether resources present in the resource pool500are adequate for use in a given scenario, an inventory of the capabilities of the resource pool as a whole500and/or of its constituent resources in particular501,502,503, may be queried and listed in the course of operation of an embodiment. Such an inventory of resources' capabilities can be obtained, for example, by inquiring the hardware and software using appropriate API function calls or via an equivalent process such as deploying a specific test case consistent with a use-case to validate. Such resources as codecs native to one or more particular libraries, such as FFmpeg, or NVIDIA's CUDA, in addition to external library wrappers, and software components for bridging said libraries to accelerating hardware can populate the resource pool500.

Capabilities thus inventoried and enumerated should likewise be understood. In a further embodiment, such capability inventory information may be coupled and integrated with error handling routines equipping said embodiment to gracefully recover from cases where an input media stream or processing task cannot be carried out by said embodiment for lack of able processing resources (including but not limited to absent media codecs), inadequate licensing, or due to any other limitation. In embodiments of the present invention in which a resource monitoring engine400is present, the result of such capabilities inquired of the resource pool500may be included within the load monitoring information450inquired by said resource monitoring engine400, and subsequently provided by said engine400to said attribution module600as part of a resource utilization data query640.

In a further implementation, the resource pool500can be configured, as further described herein, to allow for a shifting of processing away from one principal resource type and toward another. For example, a toggling, or even a permanent switching away from GPU or dedicated hardware (using, for example a CUDA-controllable resource) to CPU or general purpose hardware (using, for example, an FFmpeglibavcodec resource) is envisioned.

Such a processing changeover, for example, from a specialized or general-purpose hardware-based resource (e.g. GPGPU) to a CPU-resident resource, can provide a particularly desirable alternative processing means in cases where stream parameters or encoding particularities for a specific input media stream101change. Such stream parameter or encoding changes are for the most part unexpected and unwelcome. Furthermore, in certain implementations, they can prove particularly unwelcome to a security system video monitoring workstation on which a large number of streams containing encoded or compressed video are processed. This is especially the case when such parameters change in the course of streaming or transfer of a stream101itself, as when the stream101transitions from a fully standards-compliant encoding to a noncompliant (or to an otherwise substandard or lax) adherence to a particular encoding specification.

In such cases, a GPU-based resource is typically unable to properly carry on the processing tasks a stream whose encoding parameters change mid-stream in an uninterrupted and seamless manner. This often flows from GPU libraries' tendency to be of a closed and proprietary nature, which in turn impacts on such libraries' more sharply delineated and highly specific functionality. In such cases, the specific processing resource utilization configuration previously specified by the routing command650in place is no longer adequate to successfully carry out the intended processing. Accordingly, and further to the dynamic and unexpected change in streaming parameters, the resource previously attributed by the foregoing routing command650is caused to crash or otherwise terminate unexpectedly. In such instances, a means by which to recover from the preceding incidental and dynamic change in stream parameters is required. It will be appreciated that any resource or resource-related component can be afflicted with a crash. In a GPU use-case, for instance, a resource crash can occur because as a result of any number of various independent issues. For example, a physical component on the GPU board itself can fail, a driver there attached can fail, or the decoding library (or other on-board processing resource) can likewise fail. In addition to such sources of potential crashes, changes in stream parameters such as resolution, colorspace, compression type, or encoding can likewise cause a resource—unable to handle such changes—to crash.

It will be appreciated that in many deployment scenarios, such a resource crash recovery mechanism must operate in a manner affecting the processing (e.g. decoding) of the offending input media stream101, and accordingly the operator or user experience, either minimally or as seamlessly as possible. In contrast with the foregoing often proprietary hardware-specific processing libraries having limited scope, a reattribution of the offending input media stream101as rapidly as possible to a more robust and CPU-based processing resource502, can prove invaluable in such cases. Furthermore, it will be appreciated that allocating additional CPU cores within a resource pool500in implementations where a crash recovery mechanism involving diversion from GPU-based resources to CPU-based processing resources can improve performance and overall robustness of the crash recovery mechanism itself. This is largely because additional cores can mitigate possible oversaturation of resources, particularly those whose operations mutually parallelize with little difficulty.

It will however be appreciated that contingencies in the event of failures of CPU-based processing resources should in implementations likewise be envisioned. It is possible, for instance, for a software-implemented decoding resource to crash for reasons either similar or entirely different from a possible hardware-implemented counterpart.

In deployments in which error recovery for decoder components or other processing resources is present, a crash recovery mechanism can be implemented by way of code to execute on the CPU. Such code can include algorithms by which a decision as to which resource(s) should be used to carry out a specific processing task. Furthermore, the algorithm can include measures by which to transition to various resources, such as when either the driver API returns an error code, or alternatively when some abnormal behavior is detected (such as when a resource remains in an indefinite state for an extended period of time). While it can technically feasible to implement them within one or more GPUs, it will be appreciated that in implementations, all of such recovery mechanisms or portions thereof—including resource allocation, execution, monitoring/detecting possible error conditions, and deallocation—can advantageously operate on one or more CPUs. Thus, as specific details pertaining to GPU implementations of resources can remain opaque to implementers of embodiments described herein, decisions can be made by algorithms based on the CPU, with API calls implemented by the driver530.

It will be further appreciated that the individual processing resources501,502,503that make up the resource pool500may, in various embodiments of the present invention, benefit from being mutually accessible resources. In such cases, mutual accessibility is beneficial, as it facilitates sharing or otherwise directing the outputs of one processing resource502toward the inputs of another processing resource503. Such resource accessibility provides a wide array of potential resource sharing policies. In a further embodiment, a more complex resource accessibility policy may be defined or configured to variously restrict, encourage, optimize, or limit the use of any one or more processing resources501,502,503in accordance with the scenario in which said embodiment is deployed.

The deployment of certain resource sharing policies to the resource pool500may effect—whether implicitly or explicitly—aloud balancing policy. Indeed load balancing may constitute, in at least certain embodiments of the present invention, a key benefit. More sophisticated load balancing policies may encourage the attribution of processing tasks to any one or several processing resources501,502,503. In an embodiment where a processing task to perform is the result of several independent (or at least divisible) subtasks, such execution decisions as splitting, forking, or merging portions of data among multiple resources501,502,503may be envisioned. In a further embodiment, and particularly in cases where processing urgency, priority or other time-sensitive considerations are of primary importance, a processing load may be may be assigned, to the greatest possible extent, among multiple resources501,502,503.

In a distinct scenario, a processing error or failure using a specific resource501, including but not limited to a GPU-based codec, can occur. Numerous possibilities of failure or crash of any one or more resources within the resource pool500are possible. In implementations, these may be inquired using the inquire capabilities of any one or more component libraries' software API function calls. Likewise, a driver530or operating system bug, data corruption, or an unguarded attempt to instantiate too great a number of resources on the GPU can cause the failure or crash of other resources. The occurrence of such errors, particularly in mission-critical environments, can occasion the need for a rapid recovery whereby such processing is taken over by a CPU-based processing counterpart instead as described herein. Specific crash recovery strategies can be elaborated, particularly when previous crash heuristics are known or are otherwise available. For example, if a particular compression or encoding format observed or detected within a stream is known to trigger or otherwise cause a crash of a particular (e.g. GPU-based) processing resource501, such as a video decoder, as well as a more robust (e.g. CPU-based) processing resource502, also for video decoding, strategies such as skipping decoding one or more image frames with such a compression type or encoding format can be skipped, or some other exception handling routine can be triggered. It will be appreciated that a particularly verbose API library with comparatively exhaustive inquiry capabilities can prove invaluable both in identifying such operational anomalies, as well as in handling recovery operations thereafter.

Once a result is yielded by a processing resource501,502,503, said result may in some embodiments be made readily available externally999. The possible nature of such results is to be appreciated here in a large and widely encompassing manner. Processed results may be understood as being, in a non-limiting enumeration, numerical values, images, or any portions of partially or fully decoded video.

In another embodiment, data generated or otherwise output by one or more individual processing resources may be in a raw output form relative to said data's final purpose. In such cases, said result data may likewise not be in a usable, contextually intelligible, or otherwise useful form. In such an embodiment, it may be preferable to await the availability (from within the resource pool500and/or without) of all contextually relevant results before undertaking an encapsulation process appropriate to the deployment scenario of said embodiment. Said encapsulation process (including but not limited to activities involving or relating to stream encoding and/or transfer encoding) may occur, in some embodiments, subsequent to the production of said raw outputs and entirely within said resource pool500. In other embodiments, portions of said encapsulation may occur partially or entirely external to the resource pool500. In a further embodiment, a variable approach to such raw output data form may be envisioned.

Individual processing resources501,502,503, or indeed the resource pool500collectively, may communicate with the resource monitoring engine400in embodiments in which the latter is present. When the resource monitoring engine400is absent, the resource pool500communicates with the attribution module600. Discussion of aspects regarding the assignment of a processing load is further made in the attribution module600section herein. Likewise, discussion of aspects regarding stream and/or transfer encoding is further made in the output module800section herein.

Attribution Module

Before execution of the main processing steps can begin, embodiments of the present invention must bridge the input data received with the capabilities of processing resources present. As discussed in the previous sections, this coordination endeavor requires several steps.

Once the input media streams to process have been received by the input module100, and their nature has been ascertained, as well as the specific system configuration and particular instructions with which to operate, in addition to a due characterization of the processing resources and information about their suitability for the processing tasks has been obtained, embodiments of the present invention may proceed to assign the necessary processing tasks to the processing resources present. For example, the FFmpeg decoding library's avcodec_find_decoder( ) and avcodec_find_decoder_by_name( ) functions can prove useful in locating specific decoders to decode specific input streams101,102,103. Such codecs may non-limitingly include those enumerated at <http://ffmpeg.org/doxygen/trunk/group__lavc__core.html #gaadca229ad2c20e060a14fec 08a5cc7ce>.

This dynamic sets forth the motivation for the major decision-making component central to embodiments of the present invention. The role of the latter component, more abstractly referred to as the attribution module600, balances the operational and processing needs of incoming data with the system's ability to accommodate such processing with little perceivable delay. In embodiments of the present invention, the tangible result of such decision-making is the determination of one or more routing commands650which express the explicit association of available resources501,502,503to input streams101,102,103to process. The determination of said routing commands650is typically an ongoing exercise that in a further embodiment of the invention takes into account a collection of elements that includes knowledge of the precise nature2500fthe demand100and supply6400fall available processing resources500, in addition to externally articulated policy350. Likewise, in a further embodiment of the invention, the generation and issue of a routing command is the result of the aforementioned elements effectively cooperating as a control system to both respond to and govern the system's processing needs.

As discussed previously herein, in various embodiments of the invention, the input module100typically contains a changing number of input media streams101,102,103as these are variously received and subsequently handled by the system over time. Determining the individual nature of said streams101,102,103(including, for example, type, framerate, and resolution) is one of several important elements in the generation of a routing command650. This is particularly due to the fact that various resources may be present in embodiments of the present invention, with each of said resources being suited to a finite number of processing tasks typically required by the various streams in the input module100. The aforementioned determination of stream type is one important piece of information to be considered when deciding which one or more available resources500are best suited to handle which stream100. In a further embodiment of the present invention, the attribution module600may compile stream type information consisting of stream payload analysis results250received for streams in the input module100over a given period. While a format intelligible to said embodiment is sufficient, said compilation may in a further embodiment take the form of a list of stream type information601. In another embodiment, such information may require standardization (including but not limited to XML validation) following collection prior to being rendered intelligible, standardized, and usable.

Likewise, the attribution of resources500to input streams100may be subject to a similar approach in which said resources'500capabilities—both in terms of technical suitability for a given processing task as well as their temporal availability to carry it out—are monitored. The recurrence or precise period of such monitoring may vary in accordance with the scenario and implementation requirements of embodiments of the present invention. Said suitability and availability are among the basic information requested and returned in a resource utilization data query640. The attribution module600may likewise include a pair of modules to manage and implement said resource utilization data query640; in a further embodiment, a resource query module607may manage all such queries640made to the resource monitoring engine400, while information correspondingly received from such queries may be periodically collected in a resource table module602for further consideration by the attribution module600. It will be further appreciated that, owing to the changing operational loads of various resources in the pool500, the resource query module607may in various embodiments issue a query640in fairly regular time intervals.

While the resource monitoring engine400and its associated monitoring450and querying640(607) activities may be explicitly absent from some embodiments, the resource table module602may in such embodiments be adapted to include complementary information as the maximum number of streams that each resource501,502,503may be permitted to accept at any one time, with said resources' busy/free status gleaned not through polling or other observation, but instead deduced through calculation. This latter scenario may be encountered in a further series of embodiments, particularly mission critical environments in which conservative attribution of resources is a key consideration, or in other implementations in which resource utilization follows a fairly predictable pattern of input stream100traffic.

In a manner analogous to the standardization previously discussed with regard to the stream type information601listing, the input configuration and instructions603received, typically from the operator input module300when it is present, or from any other external source with requisite privileges, may likewise undergo a similar validation process to ensure that their contents as expressed are intelligible to the attribution module600. In a further embodiment, the attribution module600may be equipped with an instructions parser608to ensure that any translation required, such as between a third-party human interface device and/or other user interface module which may form a part of the input module300, is performed before input configurations and instructions603are provided to the comparator604—the main decision-making module within the attribution module600—whose components and operation will be discussed presently.

In various embodiments of the present invention, the comparator604receives the stream type info601, in addition to data from the resource table module602, and the input configuration and instructions608to formulate a decision as to which input stream101,102,103to assign to which resource501,502,503, with said assignment ultimately formalized as a routing command650. Each of the former elements601,602,608supplied to the comparator604originates in some form from the environment in which an embodiment of the invention operates, and for this reason such elements may collectively be appreciated as operational stimuli. To ensure that decisions by which input streams100are matched with processing resources500deemed optimal for a given implementation or scenario, embodiments of the invention apply a series of weightings to various raw operational stimuli which may vary with time. The routing command650to generate thus follows an ongoing statistical analysis of specific qualities and quantities contributed by said operational stimuli. In a further embodiment of the present invention, such statistical analysis may include the application of fuzzy logic to various operational stimuli. In a still further set of embodiments, said numerical values may additionally feature, for at least some parameters, basic or default set points or default/reference values.

In certain embodiments, the input configuration and instruction scores calculator610assigns numerical weights to certain input configuration and instructions350,603received and duly interpreted (in scenarios and/or embodiments where such interpretation is necessary) for said embodiments by the instructions parser608. Such numerical weightings provide a quantifiable measure of the importance, usefulness, relevance, severity, and/or priority to assign to specific types of configuration information or operational instructions. In a further embodiment, said weightings may be at least in part assigned as a consequence of a set of artificial intelligence algorithms. In an example embodiment, the severity and priority to provide to an emergency stop command issued by an operator may thus take precedence over another setting whereby said embodiment is set to normally prefer a hardware processing resource configuration, which may in turn have higher priority than a specific system preference for one of two otherwise identical resources during minimal load periods. In a further embodiment, the scores assigned are at least partially the result of a previously-specified qualitative or quantitative value (or a set thereof), said value(s) specified to said embodiment. Said value(s) may be specified through such channels as an operator-accessible interface or by personnel responsible for deployment or maintenance of an embodiment of the present invention. In a further embodiment, multiple aforementioned weightings may be combined via or during operation of said instructions scores calculator610. In a still further embodiment, said values may be grouped into like-themed categories, either explicitly by deployment personnel, or through an algorithm present within the input configuration and instructions scores calculator610itself.

In various embodiments, a similar weighting, score calculation and attribution approach may be applied by the stream scores calculator and sorter612to the stream payload analysis result250subsequently converted into stream type information601. Such quantification may be useful, for example, in cases where incoming streams to said embodiments require or otherwise benefit from ranking or similar quantification for purposes of determining or extrapolating operational considerations such as the respective priority to assign to an input stream101,102,103, or an optimization heuristic to apply which may be a function of the amount of data to be processed (which may in turn depend on framerate and/or resolution). Said quantification is, in various embodiments, partially a result of scores—and which may specifically concern or otherwise incorporate weightings regarding input instructions/configurations and streams—the latter of which may be calculated, result from, and received from the input configuration and instructions scores calculator610. For example, stream type info601supplied to the stream scores calculator612might reveal that a specific input stream101has a specific framerate which might exceed some maximum threshold permitted by previously specified input configuration and instructions603. Accordingly, said maximum threshold may be weighed for relative importance and consideration and be given a particularly high score by the input configuration and instructions scores calculator610. Furthermore, the stream scores calculator might provide a weighed value for said stream101indicating that said stream's framerate should be subsequently throttled or minimized in a later processing stage.

Likewise, the aforementioned weighting and quantification done by the stream scores calculator and sorter612may further be particularly valuable in in the absence of explicit instructions, whether externally supplied603or parsed608, regarding the handling of specific input streams. The stream scores calculator and sorter612may, in a further embodiment, calculate values which may complement or otherwise supplement those values originating from the stream type information601. In a still further embodiment, said scores may be operatively combined with an artificial intelligence algorithm and be calculated, recalculated, and recombined to on an ongoing basis. Once weighted values have been assigned and associated to the supplied stream type information601, the module's612further ability to prioritize the needs or one or more operationally relevant particularities may in a still further embodiment prove valuable. It will be appreciated that continual review, recalculation, and updating of said data612may be particularly useful in embodiments where rapid and/or numerous changes in the input module100are typically observed. Moreover, values for which a weighting is calculated or assigned may be grouped into like-themed categories, either explicitly by deployment personnel, or through an algorithm present within the input stream scores calculator and sorter612module itself.

In a manner analogous to the foregoing, a similar weighting, score calculation and attribution approach may in various embodiments be applied to results of a resource utilization data query640further and intelligibly converted and compiled in the resource table module602. As was the case previously, the resource scores calculator and sorter614may receive scores calculated by the input configuration and instructions scores calculator610and which are of particular concern or consideration to the attribution of processing resources. For example, a previously-specified603preference for hardware processing resources received from the set of input configuration and instructions scores calculator610may be further combined with knowledge of the busy/free status of all (or specific) hardware resources by the resource scores calculator and sorter614. Further to such combination, a further score may be complementarily calculated; in this case, the weighting expressed by said score would advantage specific hardware resources rather than otherwise equivalent software ones. It will likewise be appreciated that in accordance with the changing utilization and operational loads of the various resources in the pool500, the determination of said scores by the resource scores calculator614would, in a manner analogous to aforementioned score modules610,612be undertaken and accordingly refreshed at intervals compatible with deployment and operational scenarios of respective embodiments of the present invention.

Additionally, and in a manner akin to the two previously described modules, parameters for which a weighting is calculated or assigned may be grouped into like-themed categories, either explicitly by deployment personnel, or through an algorithm present within the resource stream scores calculator and sorter614module itself.

The two broad parallel weighting and score calculation processes just described—namely those involving, on the one hand, the input scores calculator FA and the stream scores calculator612, and the input scores calculator610and the resource scores calculator614on the other—implement a weighting attribution and first-round prioritization process involving their respective operational stimuli. This cascaded score computation follows the importance of determining the course of operation of embodiments of the present invention as a function of input supplied by human operators and/or deployment personnel while simultaneously subjecting said operation upon a current view of said embodiments' processing capabilities and raw processing needs. Once the scores and weightings based on operational stimuli for input streams101,102,103to handle612and available resources501,502,503with which to handle said streams614have been independently determined, the two sets of scores are brought together in the scores comparator and merger616module.

It will be appreciated that the two homologous sets of weightings provided612,614represent statistically optimal rankings for the specific scenario in which an embodiment of the invention is deployed. Thus, the weightings and scores emphasizing the processing needs of input streams are blended with weightings and scores emphasizing the available processing capabilities with which to handle said streams. In a further embodiment of the invention, the scores comparator and merger616proceeds with a first attempt at matching the aforementioned stream scores612with the resource scores614by examining those with as many overlapping scores occurring in as many overlapping categories as possible.

Once the two aforementioned sets of numerical data612,614are joined in the scores comparator and merger616module, a prototypical series of potential matches is assembled. In an embodiment of the invention, such assembly is typically carried out by iteratively attempting to combine or otherwise fit the highest-ranking respective scores of each of the aforementioned data sets together. Once a number of highly-ranking candidate fits or combinations of said merged data has been assembled in the scores comparator and merger616, in which one or more streams101,102,103is thus provisionally matched for processing with one or more processing resources501,502,503, said provisional combinations are made available to the routing command engine618.

It will be appreciated that the time interval(s) with which the score calculators610,612,614,616described herein operate, calculate, regenerate, and apply said scores and attendant sorting/priority information may likewise vary as a function of the needs dictated by operational scenarios of respective embodiments of the present invention. In a further embodiment, these respective time intervals may be set independently for each of the aforementioned score calculators. It will be appreciated that in all cases, it is advisable to ensure that potential vulnerabilities resulting from lack of data freshness during the respective collection and integration of such scores is sufficiently abated as to prevent inadvertent or otherwise harmfully erroneous operation of respective embodiments of the present invention. It will be further appreciated that the precise measures to take, including but not limited to the adjustment of the aforementioned time intervals, will vary in accordance with the needs and operational scenarios of said embodiments.

The routing command engine618receives all provisional combinations assembled by the scores comparator and merger616module. In an embodiment of the invention, additional error checking or validation may be integrated into the routing command engine618as it proceeds to discard any provisional combinations that may be considered invalid, undesirable, or impracticable in light of specific operational or policy-related conditions. In another embodiment, such discarding may be sidestepped by selecting the provisional combination having the highest calculated weightings in as many categories as possible for a given embodiment within a given scenario. Once any and all provisional combinations deemed non-optimal or not desirable are definitively eliminated, the routing command engine618generates a formal routing command650in which the specific resource(s)501,502,503to process one or more specific streams101,102,103is expressed.

In an embodiment of the invention, the issuance of a routing command650by the routing command engine618marks the final operation step of the comparator604, with said routing command650being received by the dispatching module700.

In some deployments, heuristics such as those described herein can be collected and analyzed to further the operations of a crash recovery mechanism following the crash of a resource501,502,503, or even for broader purposes of future crash prevention involving said resources. Furthermore, specific API-based failure codes can provide the basis for identifying a potential dynamic reattribution of resources501,502,503to a specific stream101,102,103to operate a specific crash recovery scenario. Thus, while handling decoding operations for a specific stream101having a specific set of stream parameters for example, a specific GPU-based resource might encounter a crash. Prior knowledge wherein a specific hardware processing library element used in combination with a stream of a given type is likely to result in a crash can be advantageously exploited such that in future occurrences, said library element and stream types will not be matched by the attribution module600. In a further implementation, an amended routing command650′ can accordingly be generated immediately to implement such reattribution.

Dispatching Module

The dispatching module700implements750the stream-to-resource attribution specified by routing commands650issued from the comparator604. In a further embodiment, a plurality of routing commands650may be issued in a burst to the dispatching module700, which may temporarily store said plurality of routing commands650in a routing command queue605. Such temporary storage of routing commands650may in a further embodiment be governed by a control structure consisting of wait statuses applied to various routing commands650in the routing command queue605until a specific condition is satisfied and wherein bulk number of routing commands650may be issued. In another embodiment, no such queue605need be present, with routing commands650received by the dispatching module700being provided directly to the sorting network701. The latter module701may in an embodiment be implemented by way of a switch fabric which in an embodiment of the invention allows routing of input module100contents to the resource pool500. In another embodiment, a semaphore or other control system adapted to and commensurate with the operational scenario of said embodiment may likewise be envisioned for said sorting network701in lieu of a switch fabric.

Routing commands650are typically issued until either no additional input media stream101,102,103is received in the input module100, or when all processing within the resource pool500has completed. It will be appreciated that a key effect of the dispatching module's700operation is the latter's ensuring that no contention for any processing resources501,502,503present in the resource pool500is encountered.

Although operational stimuli ultimately originating from the operator input module300, the input module100, and the resource pool500may be correctly conceived as forming the basis of routing commands650generally, in a further embodiment, a new routing command650′ may be issued further to the fulfillment or completion of a previously-issued routing command650. An amended routing command650′ may be desirable or necessary, for example, in cases where a processing resource501deemed best suited for a particular processing task required for a specific input media stream101was not initially available (e.g. not present, not accessible to an embodiment of the invention, or accessible but occupied with processing tasks for a different media stream, etc.) for said stream101, but later becomes available. In such cases, an embodiment of the present invention may deem it desirable to interrupt the processing resource502at the earliest opportune moment, relieve said resource502of its processing duties for said stream101, and transfer the remainder of a specific processing operation for said stream101to the newly-available processing resource501best suited for said task. In such cases, the amended routing command650′ is issued by the attribution module600using a process in many ways similar to that described previously. However, the generation of an amended routing command650′ occurs as a result of the resource scores calculator and sorter614located within the attribution module's600comparator's604receiving an update602regarding the recent availability of the processing resource501determined in the present example to be better suited than the originally-attributed resource502. Thus, the tenure of any one specific routing command650may be cut short by way of an amended routing command650′.

In a different scenario, and in a further embodiment, an amended routing command650′ may be issued to revoke the tenure of a particular processing resource502—even if it is indeed the best suited resource available to perform a given task on a given input stream102. This might be desirable in cases where a new input stream101appears in the input module100for processing and for which said processing resource502is likewise the best suited of all available resources. In such an example, the new input stream101may suspend the processing privileges502previously afforded to the previous stream102because the new stream101is specified as having (or otherwise known by said embodiment to have) higher priority or urgency than the input stream102previously being handled by said resource502. Such “bumping” or reassignment of previously allocated resources by way of an amended routing command650′ may be effected for any reasons and/or criteria deemed appropriate by the specific scenarios in which specific embodiments of the present invention are deployed.

Accordingly, in circumstances unrelated to load balancing alone, a reattribution of resources to streams by way of an amended routing command650′ can operate in scenarios pertaining to recovery following a crash of a particular processing resource501,502,503. As described herein, such resource reattribution can be particularly important in cases where one or more stream101,102,103parameters change—especially without prior warning—in the course of streaming. In such cases, such changes can result in oversaturation of available resources within the resource pool500. And as discussed herein, further to such changes, a specific processing resource501,502,503, previously attributed by way of a routing command650,650′, can be unsuited and thus unable to carry on with the intended processing operation. This can occur, for example, in cases where an input media stream101,102,103is a video stream from a security camera whose vendor has deviated from a particular standard and wherein the stream encoding parameters change unexpectedly from one moment to another. In such cases, a further amended routing command650′ can be generated, such that processing of the offending stream101whose parameters have changed unexpectedly as to cause a crash of the processing resource501can be rerouted to a better suited processing resource502. Rerouting can, in an example divert processing from an originally-attributed processing resource501(e.g. a hardware-specific or GPU-based decoder, such as might be offered with NVIDIA's CUDA) to another processing resource502(e.g. a general purpose CPU-based decoding library, such as FFmpeg). It will be appreciated that identifying the most appropriate resource(s) available within the resource pool500to which to potentially divert such processing in light of the unanticipated change in stream parameters can be determined, with the appropriate modifications, by combining any one or more of the various means previously described herein. In such implementations, operation of the stream detection engine200can be modified to specifically (re) detect the parameters of an offending stream101, with the updated stream payload analysis result205accordingly being provided to the stream type information601listing. The aforementioned measures can in some embodiments additionally combined with buffering a portion of input media streams101,102,103so as to mitigate the potential for loss of stream data. In further implementations, such buffering can be further coupled, where technically feasible, with a more sophisticated and robust control mechanism to request retransmission of stream packets known to have been lost or dropped as a result of resource saturation.

In a further embodiment, a routing command650may incorporate any combination involving one or more input media streams101,102,103and any number of processing resources501,502,503. This is particularly relevant in cases where multiple (but separate) processing operations must be carried out on one or more input media streams101,102,103. A corollary to this, and an equally valid scenario, is one in which a specific processing operation to be carried out by a single resource501requires multiple input media streams101,102,103.

Output Module

In cases where the processing operation(s) to perform on one or more input streams101,102,103within an embodiment of the present invention would result in a modification to the content of said input streams, one or more new result streams are generated by said embodiment. This is done so as not to destroy (or otherwise destructively overwrite) the original input stream101,102,103received. Such generation is typically required when the result of said processing500is also a media stream. In a further embodiment, the latter new media stream, also known as an output stream801, is typically packaged in a format identical or similar in composition or structure to the originating stream. In a further embodiment, and particularly in cases where the output stream is a combination of multiple types of streams, or a derivation into a more limited subset of data, an optimal output stream packaging, composition, and structure (including but not limited to transport type) into which to format the resulting data is determined and selected by said embodiment. In a still further embodiment, said selection is based on maintaining similarity with said target data's constituent data, which, it will be appreciated, is a function of modifications applied by the resource pool500, which includes without limitation, any merging or stripping of various data performed on original data streams101,102,103. In another embodiment, the format of the output media stream801may be a function of previously-specified configuration or instruction350data. In a still further embodiment, said formatting of output media streams801depends on the availability of target codecs as well as the embodiments overall capability to encode in said format.

In another embodiment, the input stream data integrity conservation principle described above may be waived either in whole or in part. In the latter case, said waving may be implemented on some conditional criteria, specified in said embodiment as part of the configuration data and instructions350supplied to said embodiment. In a further embodiment, no input stream101,102,103is conserved but rather is acted upon directly by one or more resources501,502,503in the resource pool without any prior data copying.

Once the output media stream801is generated or otherwise made available in the output module800, it may be rendered available for use by being output to specific locations, including but not limited to storage media or to a URL accessible by an embodiment of the present invention999. In another embodiment, said output module800may be entirely optional, with the raw results of processing pool500being made directly available to said storage media or accessible URL999.

In a further embodiment, particularly one in which copies of the streams101,102,103are fetched and placed into the input module100, such input stream data integrity conservation may be viewed as a still further option that is desirable in an even more limited series of scenarios.

Display Management Module

A further module that may be optionally present in certain embodiments is the display management module900. In certain settings, embodiments of the present invention may be accompanied by a screen or other visible area or surface that enables a human operator, such as a guard sitting at a security desk, video wall, or video monitoring workstation, to view the one or more input101,102,103and/or output801media streams available to embodiments of the present invention. In a further embodiment, said display management module900,900′ may present additional media streams and/or other data fetched from an external source099, but which have not required fetching and placing into the input module100for subsequent processing. The display of such streams may be implemented by way of a GPU dedicated for display purposes and set up to arrange said streams in a multi-tile or multi-window layout, said layout and arrangement selection being provided to the display management module900by way of one or more display signals received by the latter module900by said dedicated display GPUs. In a further embodiment, if said additional media streams require processing by said embodiment of the invention, said streams are fetched from their original locations099and made available to the input module100, as discussed herein. Streams not requiring processing may likewise be fetched099separately and shown in the display management module900.

In a further embodiment, the display management module900may allow one or more (typically human) operators to specify a particular set of instructions. Said instructions, otherwise known as operator input950, may be issued by any HID or HID-like means, such as through a touch screen monitor on which aforementioned input101,102,103and output801is displayed. Said instructions950are transmitted to the operator input module300, previously discussed herein, for handling by said embodiment of the present invention. In a still further embodiment, said operator input950may be limited to specific tasks, including, in a non-limiting enumeration, decreasing the framerate of a given video stream, pausing a stream, specifying a region of interest within a given stream for purposes such as zooming, specifying full-framerate decoding of one or more high-resolution streams, and other image processing tasks.

In a still further embodiment of the invention, the display management module900may be subject to various levels of user privilege and/or accessibility for various human user types. Without limiting the foregoing, levels might include such profile types as non-existent (where no access to any display content is granted, for example), spectator (with view privileges only), limited functionality, advanced functionality, superuser, wherein each successive user profile grants additional privileges and/or access to said profile holder, which in turn influences the extent of the operator input950that said profile holder might supply. In another embodiment of the invention, the display management module900may be associated with the aforementioned operator input module300. It will be appreciated that user or operator input to signal the occurrence of anomalous operation of any one or more streams displayed can be accepted. Such input can be useful, for example, to signal to administrative personnel the occurrence of an apparent crash of a component or other performance-related issues. In another embodiment, the two modules300,900may be entirely (including physically) distinct from one another.

An important element in crash recovery capabilities as described herein concern the possible failover qualities discussed for various deployments. A similar element in the crash recovery process concerns the quality with which such failover is achieved. The seamless handling of decoder or other resource crashes is a similarly valuable quality to a resource crash recovery implementation. Moreover, in many deployments, a failover implemented in such manner as to effect a minimally user-perceptible switchover is envisioned. Furthermore, handling of such resource-related abnormalities, whether crashes or other abnormal terminations, is carried out such as to minimally impair or affect the functionality of the resource impacted by a crash.

Thus, a crash of a resource501such as a GPU-based decoder in the course of decoding an input stream101can result in said input stream101being decoded by a CPU-based decoding resource502with minimal turnaround time. In this hypothetical example, input stream101can be processed into output media stream801. Output media stream801can accordingly be displayed within a display management module900, occupying all or, in a further implementation, a tiled portion thereof.

Crash recovery mechanisms as described herein can accordingly and individually target any one or more single output media streams801displayed within a display management module900. Accordingly, a single output media stream801can occupy a specific tile within a display management module900. In the case of an allocated processing resource's501crash or other operational abnormality, processing of the input media stream101corresponding to the output media stream801occupying a single tile can be restarted. In implementations, restarting of processing can involve the allocation and attribution of a second processing resource502to the input media stream101. The foregoing can be implemented such that visible and/or human perceptible interruption to the output media stream801can be minimal or even imperceptible. Following an amended routing command650′, processing of the input media stream101can resume with the second processing resource502, potentially a CPU-based one, carrying on with processing. Playback or live streaming of the corresponding output media stream801within the display management module900can, to a human operator, continue to appear within its previous position or allocated tile space to have been minimally impaired or otherwise affected by any adverse operational issue. As described herein, measures such as buffering techniques and/or other dejittering methods can be advantageously used to minimize observable video or image degradation, noise, or other undesirable artifacts.

ADDITIONAL CONSIDERATIONS

Further consideration of various aspects of modules involved in various embodiments of the present invention may be given. For example, various components of, and indeed the modules described herein and which make up embodiments of the present invention need not be physically contiguous nor be within a proximate geographic area. In some embodiments, various modules or aspects thereof described herein may be connected to or within said embodiments by way of some network connection, such as a LAN or the internet. In a similar vein, the implementation of various aspects, components, or functionalities described herein as being implemented within a specific module or submodule need not be interpreted as being necessarily limited, whether in whole or in part, or for any embodiment of the present invention, to said modules as disclosed herein. Likewise, any or all of the aforementioned modules may be implemented, in whole or in part, in one or more distinct computers. Finally, modules described herein and connected through a network may be connected among said one or more computers through any networking technology, topology, without limitation.

FIG.11shows an example partial layout of an embodiment involving the processing steps described herein in the case of hardware-based processing resources. A board using the Maxwell microarchitecture can non-limitingly be envisioned for implementation in such cases. Packets for an input stream are received from a source, such as a storage device, media, or URL099, and must be reassembled, and interpreted, such interpretation having been discussed herein in the stream detection engine200section. Such detection can, among other things, serve to determine and identify the nature of an input stream101. This can include determining the encoding standard (such as H.264 or H.265) with which the input stream101complies. It will be appreciated, as discussed herein, that parameters or elements of the input stream101can change dynamically and unexpectedly, and that the nature of the new parameters must likewise be detected for purposes of properly recovering from (or averting) a processing resource501,502,503crash.

A critical processing step, for certain embodiments discussed herein, is the decoding501″ of an input stream101. In an implementation, a video decoder510module can communicate with various different hardware and/or software resource implementations. Such implementations can non-limitingly include processing resource libraries, which can non-limitingly include FFmpeg's AVCodec, NVIDIA's CUDA, and Intel's QuickSync. Implementations can further communicate with any one or more GPU-based or CPU-based layouts configured to communicate with and/or operate using any one or more of the foregoing processing resource libraries.

Communication with any one or more of the foregoing types of processing resource libraries can involve the use of API calls offered by said processing resource libraries, which operate within the driver. InFIG.11, the CUDA library block is non-exhaustively illustrated as communicating with an NVIDIA driver530; other implementations can include a similar communication and interoperation structure within any one or more other processing resource libraries. Use of the foregoing API calls enables interaction with the hardware or software processing components which can populate the resource pool500. A CUDA-based implementation can for example leverage a software object known as an NVCUVID decoder520, of which one or several instantiations can be made. The NVCUVID decoder520is itself a creature of the NVCUVID API, a specific application programming interface allowing the decoding of video on CUDA-based GPUs. The NVCUVID driver, is itself implemented by the API provider (NVIDIA) within said provider's driver530. A high-level API-based interaction with the GPU typically obviates the need for integrating parties to be aware of operation or implementation details of such codecs or other processing resources within a GPU.

It will be appreciated that NVAPI and the CUDA API itself can both be used by the NVIDIA driver for compatible GPU boards. It is possible to inquire the resource capabilities, for example, of several Maxwell architecture-based NVIDIA GPUs, using NVAPI, an application programming interface provided by NVIDIA to directly access and control core elements of NVIDIA GPUs on Windows platforms. Such resource capability inquiries can non-limitingly allow general identification such as version determination and driver identification. In an embodiment involving a compatible NVIDIA GPU, for instance, it can be possible to instead make direct use of the CUDA API for at least a portion of the latter's functional breadth rather than rely on software or API-based inquiries of fundamental processing capabilities of the GPU itself. For example, an active attempt to test a possible operation of a specific processing step could optionally or likewise be made. It is likewise envisioned that in deployments it might not be feasible to implement or otherwise replace all API functionality with a single API, such as CUDA. For example, CUDA does not provide access to dedicated H.264 and H.265 decoding resources (for which purpose NVCUVID is useful), owing to the fact that CUDA is an API for programming CUDA cores. CUDA likewise does not provide access to performance counters and other statistics provided by NVAPI. Thus, in implementations, a combined or concerted use of multiple APIs can not only prove advantageous, but potentially even necessary. It will be appreciated that other hardware and software vendors offering varying technologies can likewise be envisioned for operation within a comparable paradigm. While current operating system paradigms limit allocation and use of a single driver at a time for a specific GPU board, for instance, it will likewise be appreciated that multiple libraries can be used to inquire or otherwise interact with driver. In the case of NVIDIA graphics cards, Applicant has successfully implemented crash recovery mechanisms for use with cards having a compute capability of or greater than 2.1 and with at least 512 MB of VRAM. Similarly, Intel-based solutions require that a CPU support the latter vendor's QuickSync technology.

It will be appreciated that the GPU board driver supports a specific version of a parallel computing platform and related API(s). Thus, for example, the NVIDIA driver530can support a specific version of CUDA, said version being dependent on the capabilities of the GPU itself. Capabilities are likewise related to the hardware available on the GPU to perform specific operations, whether dedicated or otherwise, as well as the APIs' own abilities to exploit such resources and operations. Version-related particularities, such as operational limitations, incompatibilities, and even performance improvements, can be introduced by or further result from changes in the versions of platforms, such as CUDA, used on a GPU. For example, on GPU boards implementing the Maxwell microarchitecture, a sevenfold increase (from 2 to 14) in the number of high-definition stream101decoded has been noted with a migration from CUDA version 3.5 to version 5. It will be appreciated that in addition to more optimized CUDA instructions, such an increase can also be attributed in part to such GPU boards having additional cycles and memory available to carry out decoding specific to the H.264 or H.265 types. The existence or exploitation of specific dedicated resources, such as a hardware component to more rapidly and efficiency carry out such processing, can also provide significant performance improvements.

Despite the foregoing, as discussed for various implementations herein, GPU-based decoding and other processing can be limited or otherwise result in poor decoding capabilities in cases of non-standard or unexpected changes in stream parameters, or oversaturation of existing resources and limited on-board resources (such as might happen when too large a number of streams is received, or a resolution or framerate of existing streams increases the load on the processing pool500).

Gracefully handling such unexpected cases can, as discussed herein, prove more flexible when a solution comprising general-purpose CPU-based resources is provided. Structuring deployments in such a manner as to minimize the frailty can decrease overall vulnerability and enhance overall robustness. The applicant has determined that implementing seamless a GPU-to-CPU crash recovery resource substitution as discussed herein, whether dynamically or as a general rule, can greatly contribute to such enhanced robustness. Notwithstanding the foregoing, it is likewise envisioned that GPU-based resources can be similarly exploited to implement failover mechanisms from crashed or otherwise abnormally terminated CPU- or software-based resources.

No explicit exclusion from the crash recovery mechanisms described herein of streams101,102,103having specific parameter characteristics is envisioned. It will nonetheless be appreciated that applying little to no discerning criteria to the parameter characteristics of input streams101,102,103deemed eligible for (or desirably suited to) such crash recovery mechanisms can prove detrimental to overall performance of implementations. Workstation particularities, including a careful survey of resources available and overall processing power, in addition to suitability assessments in light of data transfer costs between GPUs and a CPU (e.g. for low bandwidth streams) should in deployments be taken into consideration.

At the same time, it will be appreciated that many factors determine how many streams of one or more supported format sat specified framerates can be successfully decoded (or otherwise processed). Such factors include dynamic usage. Thus, additional streams should not be added to an already overburdened graphics card. Likewise, hypothetical or planned usage should be taken into account; typically, such planning should involve heuristically-derived data or benchmarks.