METHOD AND SYSTEM FOR HIGH THROUGHPUT LIVE AND OFFLINE MULTIMEDIA TRANSCODING

A versatile high-throughput multimedia transcoding station, serving a plurality of multimedia sources, employs transcoding resources including a pool of decoders, a pool of signal-adaptors, and a pool of encoders operating concurrently to realize low-latency transcoding of high-flow-rate multimedia streams. A multimedia stream contains a video stream organized into source groups-of-pictures (GOPs). Upon receiving a transcoding request indicating characteristics of a source multimedia stream and desired characteristics of a destination multimedia stream, an orchestrator rapidly allocates a resource for each GOP and coordinates activation of a content-processing assembly which encompasses the transcoding resources and means for distributing each GOP to compatible resources. The orchestrator assembly monitors progress of GOPs' processing and, when needed under high workload fluctuation, instructs a multimedia source to pause transmission. Each of the decoders, signal adaptors, and encoders comprises a respective hardware processor coupled to a memory device storing software instructions and a buffer holding intermediate data.

FIELD OF THE INVENTION

The invention relates to multimedia streaming. In particular, the invention is directed to realizing a large-scale transcoding station serving high-speed streams of diverse formations.

BACKGROUND

The rapid growth of transport capacity of communication networks enables provision of multimedia streaming at high flow rates. The constantly evolving techniques of video-signal encoding necessitates an intermediate transcoding layer to enable multimedia sources of different generations to stream to receivers which may not be equipped to decode streams of different resolutions and formats encoded at sources of different generations. In some applications, video signals are generated using codecs producing high-quality encoded signals but at a low compression ratio, thus necessitating a large storage capacity or a high-capacity communication path, of the order of gigabits/sec, to a recipient.

A traditional transcoding server is typically constructed to download a source multimedia file, to be stored locally, then produce a corresponding transcoded file which is also stored locally to be uploaded to a designated destination. The sizes of files thus produced may be of the order of a terabyte. Transcoding may be performed in a single server or cooperatively in multiple servers which may not be collocated.

There is a need, therefore, to explore methods and systems for expeditious and efficient transcoding of high-speed multimedia streams of diverse formations.

SUMMARY OF THE INVENTION

The objective of the present invention is to realize a heterogeneous distributed transcoding process integrating multiple transcoding functions while minimizing use of local storage and efficiently managing resources.

In accordance with an aspect, the present invention provides a transcoding station for multimedia transcoding. The transcoding station comprises a network interface, a pool of encoders, a pool of signal-adaptors, a pool of decoders, an orchestrator assembly, and a content-processing assembly.

The network interface is configured to receive, from a multimedia source, a transcoding request and a multimedia stream comprising a video stream organized into source groups-of-pictures (GOPs). The transcoding request indicates a first standard according to which the multimedia stream is formed and a second standard according to which an output decoded stream is to be formed.

The pool of decoders is configured to concurrently decode different GOPs, according to the first standard, to produce respective decoded GOPs. The pool of signal-adaptors is configured to concurrently process different decoded GOPs to produce respective adapted GOPs. The pool of encoders is configured to concurrently encode different adapted GOPs to produce respective encoded GOPs according to the second standard;

The orchestrator assembly is configured to assign each source GOP to an available decoder, each decoded GOP to an available signal adaptor, and each adapted GOP to an available encoder. The content-processing assembly is configured to provide each source GOP access to an available decoder, each decoded GOP access to an available signal adaptor, and each adapted GOP access to an available encoder. An output-collating module, coupled to the network interface, is configured to arrange successive encoded GOPs in the order of corresponding source GOPs.

Each decoder of the pool of decoders, each signal adaptor of the pool of signal adaptors, and each encoder of the pool of encoders comprises a respective hardware processor coupled to a memory device storing software instructions, and a buffer holding intermediate data.

The orchestrator assembly is further configured to determine, in response to a transcoding request, availability time based on current occupancy of the pool of decoders, the pool of signal adaptors, and the pool of encoders. The orchestrator assembly communicates the availability time to the multimedia source through the network interface.

The orchestrator assembly is further configured to track counts of source GOPs, decoded GOPs, and adapted GOPs waiting for assignment to respective decoders, signal-adaptors, and encoders, respectively. If any of the counts exceeds a respective prescribed threshold, the orchestrator assembly instructs the multimedia source, through the network interface, to pause transmission of the multimedia stream. When none of the counts exceeds a corresponding prescribed threshold while transmission of the multimedia stream is paused, the orchestrator assembly instructs the multimedia source, through the network interface, to resume transmission of the multimedia stream.

The content-processing assembly comprises a first content-access unit coupled to the network interface and the pool of decoders, a second content-access unit coupled to the pool of decoders and the pool of signal-adaptors, a third content-access unit coupled to the pool of signal-adaptors and the pool of encoders, and a fourth content-access unit, comprising a collating module, coupled to the pool of encoders and the network interface.

The orchestrator assembly comprises an orchestrator core coupled to three controllers. A first controller is coupled to the first content-access unit and the pool of decoders. A second controller is coupled to the second-content-access unit and the pool of signal adaptors. A third controller is coupled to the third content-access unit and the pool of encoders.

The network interface is further configured to affix a GOP identifier to each source GOP of the multimedia stream and communicate the source GOP identifiers and corresponding metadata to the orchestrator assembly and to the content-processing assembly.

The first controller is configured to queue an identifier of each source GOP and, upon locating an available decoder, instruct the content-processing assembly to process content of a queued source GOP. The second controller is configured to queue an identifier of each decoded GOP and, upon locating an available signal adaptor, instruct the content-processing assembly to process content of a queued decoded GOP. The third controller is configured to queue an identifier of each adapted GOP and, upon locating an available decoder, instruct the content-processing assembly to process content of a queued adapted GOP.

According to an implementation, the pool of decoders comprises a first number of clusters of respective distinct decoders. The pool of signal-adaptors comprises a second number of clusters of respective distinct signal-adaptors. The pool of encoders comprises a third number of clusters of respective distinct encoders.

Based on information within the transcoding request, an available decoder is selected from a compatible cluster of the first number of clusters, an available signal-adaptor is selected from a compatible cluster of the second number of clusters, and an available encoder is selected from a compatible cluster of the third number of clusters.

Thus, the disclosed transcoding station operates at a much higher speed in comparison with conventional transcoding servers and eliminates the need to download and locally store the multimedia file to be transcoded, the produced transcoded file, or intermediary format. Any transient media format is streamed from one stage of transcoding to another in a continuous way, reducing the amount of RAM memory used as opposed to storing complete intermediate media before passing them on to the next stage.

In accordance with another aspect, the invention provides a method of multimedia transcoding. The method comprises receiving, at a network interface, from a multimedia source, a transcoding request and a multimedia stream. A hardware orchestrator assembly and a content-processing assembly perform requisite transcoding processes of the multimedia stream.

The multimedia stream comprises a video stream organized into source groups-of-pictures (GOPs) and corresponding metadata. The transcoding request indicates a first standard according to which the multimedia stream is formed and a second standard according to which an encoded stream is to be formed.

Multiple decoding workers of the pool of decoding workers are activated concurrently to decode different source GOPs, according to the first standard, to produce respective decoded GOPs;

Multiple signal-adaptation workers, of a pool of signal-adaptation workers, are activated concurrently to process different decoded GOPs of the respective decoded GOPs to produce respective adapted GOPs.

Multiple encoding workers, of a pool of encoding workers, are activated concurrently to encode different adapted GOPs of the respective adapted GOPs to produce respective encoded GOPs according to the second standard.

The network interface affixes a GOP identifier to each source GOP of the multimedia stream; and communicates the GOP identifiers to the orchestrator assembly and to the content-processing assembly.

The method segments the pool of decoding workers into a first number of distinct clusters of decoding workers, the pool of signal-adaptation workers into a second number of distinct clusters of signal-adaptation workers; and the pool of encoding workers into a third number of distinct clusters of encoding workers. Based on information within the transcoding request, the orchestrator assembly selects the multiple decoding workers from a respective cluster of decoding workers, selects the multiple signal-adaptation workers from a respective cluster of signal-adaptation workers, and selects the multiple encoding workers from a respective cluster of encoding workers.

The orchestrator assembly continually tracks a first count of GOPs waiting for processing at the respective cluster of decoding workers, a second count of decoded GOPs waiting for processing at the respective cluster of signal-adaptation workers, and a third count of adapted GOPs waiting for processing at the cluster of encoding workers.

If any of the first count, second count, or third count exceeds a respective prescribed threshold, the orchestrator assembly instructs the multimedia source, through the network interface, to pause transmission of the multimedia stream.

If none of the first count, second count, and third count exceeds a corresponding prescribed threshold while transmission of the multimedia stream is paused, the orchestrator assembly instructs the multimedia source, through the network interface, to resume transmission of the multimedia stream.

The orchestrator assembly arranges successive encoded GOPs, of the respective encoded GOPs, according to an order of corresponding source GOPs of the multimedia stream.

In accordance with a further aspect, the invention provides a transcoding station comprising a network interface, and an orchestrator assembly coordinating activation of clusters of decoders, clusters of signal adaptors, and clusters of encoders.

The network interface is configured to receive from a multimedia source a transcoding request and a multimedia stream comprising a video stream organized into source groups-of-pictures (GOPs) and corresponding metadata.

The orchestrator assembly coupled to the network interface, a cluster of N1decoders, N1≥1, a cluster of N2signal adaptors, N2≥1, and a cluster of N3encoders, N3≥1.

A 1:N1selector is configured to distribute contents of successive source GOPs to orchestrator-selected decoders to produce respective decoded GOPs. An N1:N2switching element is configured to distribute the respective decoded GOPs to orchestrator-selected signal adaptors to produce respective adapted GOPs. An N2:N3switching element is configured to distribute the respective adapted GOPs to orchestrator-selected encoders to produce respective encoded GOPs. An N3:1 selector coupled to a collating module is configured to collate the respective encoded GOPs to form a contiguous transcoded video stream.

Each decoder is compatible with a first standard, indicated in the transcoding request, according to which the multimedia stream is formed. Each encoder is configured to encode an output of any signal adaptor according to a second standard specified in the transcoding request.

The orchestrator assembly is configured to track a first count of source GOPs waiting for decoding at the cluster of N1decoders, a second count of decoded GOPs waiting for signal adaptation at the cluster of N2signal adaptors, and a third count of adapted GOPs waiting for decoding at the cluster of N3encoders.

If any of the first count, the second count, or the third count exceeds a respective prescribed threshold, the orchestrator assembly instructs the multimedia source to halt transmission of the multimedia stream. If none of the first count, the second count, and the third count exceeds the prescribed threshold while transmission of the multimedia stream is halted, the orchestrator assembly instructs the multimedia source to resume transmission of the multimedia stream.

The cluster of N1decoders is selected from a plurality of decoders based on the first standard and detailed characterization of content of the multimedia stream embedded in the metadata. The cluster of N2signal adaptors is selected from a plurality of signal adaptors based on requisite signal-adaptation processes indicated in the metadata. The cluster of N3encoders is selected from a plurality of encoders based on the second standard. Each of the N1decoders, N2signal adaptors, and N1encoders comprises a respective hardware processor coupled to a memory device storing software instructions, and a buffer holding intermediate data.

TERMINOLOGY

Group-of-pictures: A multimedia stream comprises a video stream which may be organized into independent groups-of-pictures (GOPs) to enable parallel processing. The GOPs generated at different multimedia sources may have arbitrary numbers of video frames.

Content data vs. control data: The streaming data comprises content data of different forms (video, audio, text, . . . ) and control data which guide the processing and routing of the content data. The content data of a multimedia constitutes the bulk of the data of the stream.

Orchestrator assembly: An orchestrator assembly comprises an orchestrator core (functioning as a master controller) and multiple stage controllers each dedicated to a respective processing stage. The orchestrator assembly guides the work-load assignment to a plurality of heterogenous workers (defined below).

Content-processing assembly: The content-processing assembly, operating under control of the orchestrator assembly, dynamically provides to each engaged worker access to respective data.

Content-access units: The content-processing assembly is structured in the form of content-access units.

Collator: A collator is an output module of the transcoding station, coupled to the network interface, which is configured to arrange successive encoded GOPs in the order of corresponding source GOPs.

Decoder: The term decoder refers to a decompression module for reconstructing raw source data from a received compressed source data.

Encoder: The term refers to a compression module for compressing a signal-processed stream adapted to be compatible with intended receivers.

VPU: A video-processing unit (VPU), also called a signal adaptor, formats a video signal to be compatible with intended receivers.

Transcoder: The term transcoder refers to a device employing at least one decoder, at least one VPU, and at least one encoder.

Worker: The term “worker” refers to a decoder, a signal adaptor, or an encoder having a respective hardware processor coupled to a memory device storing software instructions and a buffer holding transitory data.

Transcoding station: A transcoding station is a large-scale transcoder employing a plurality of heterogenous workers where workers of any processing stage comprise a mixture of uni-functional and multi-functional workers, with the multifunctional workers comprising uni-tasking worker and multitasking workers.

Worker type: The plurality of heterogenous workers is sorted according to types of tasks that a worker is configured to perform.

Versatility indicator: A versatility indicator of a worker type is a number of tasks that a worker of the worker-type is configure to perform whether one at a time or concurrently.

REFERENCE NUMERALS

100: A system providing high throughput live, or posterior, transcoding services120: A network supporting transcoding stations140: Multimedia sources160: Transcoding station180: Clients of the transcoding system200: An overview of a structure of a transcoding station210: Dual link connecting to network120220: Network interface230: External control signals and metadata from multiple sources140231: External control data directed to Orchestrator250232: Control data directed to destinations through the network interface220235: Selector of external control data from a specific stream240: External content data from the sources241: External content data (upstream content data) directed to the pool of workers260242: Content data directed to destinations through the network interface220245: Selector of external content data from a specific stream250: Orchestrator260: A pool of workers of different types270: Dual control paths between orchestrator250and the pool of workers260280: A pool of resources communicatively coupled to the pool of workers260300: Network-interface components310: Source interaction module320: Control data and metadata extraction module330: Destination interaction module340: Source-characterization algorithm350: Source characteristics database360: Content-data-extraction module380: Transcoded GOPs rearrangement module390: Network-interface processor (or a pool of processors)400: Exemplary workers of the pool of workers260comprising integrated workers and specialized workers410: An integrated worker whereby decoding (decompression), signal processing, and encoding (compression) are performed420: A set of independent specialized workers421: A specialized worker (type-1 worker) performing decoding (decompression) of a compressed multimedia signal formed according to a specific standard422: A specialised worker (type-2 worker) performing conventional signal processes423: A specialized worker (type-3 worker) performing multimedia encoding (compression) according to a requisite standard500: Duration of transcoding functions: comparison of using integrated workers410versus using specialized workers420600: Concurrent transcoding of GOPs using integrated workers410610: A process of transcoding individual GOPs independently using multiple integrated workers410612: Received multimedia signals organized into independent GOPs614: Metadata defining structure (format) of incoming GOPs620: A process of collating transcoded GOPs in the temporal order in which corresponding incoming GOPs were received622: Transcoded GOPS arranged in proper order700: Concurrent transcoding of GOPs using specialized workers712: A process of decoding (decompressing) individual compressed GOPs independently using type-1 workers714: Conventional signal processing of decoded GOPs, received from type-1 workers, using type-2 workers716: A process of encoding processed GOPs, received from type-2 workers, using type-3 workers720: A process of collating transcoded GOPs, received from type-3 workers, to produce a stream of transcoded GOPs of the same sequential order of corresponding incoming GOPs800: Processing time intervals of independently transcoded GOPs, GOP-jT, j>0, for a case of GOPs of equal transcoding time intervals810: Indices of successive raw GOPs820: Time interval of a single transcoded GOP900: Processing time intervals of independently transcoded GOPs, GOP-jT, j>0, for a general case of GOPs of differing transcoding time intervals920: Time interval of a single transcoded GOP1000: Assignment of GOPs to a number of integrated workers410for performing concurrent transcoding processes for a case of GOPs of equal transcoding time intervals1010-1016: Transcoding tasks assigned to integrated workers410of indices 0 to 61100: Assignment of GOPs to a number of integrated workers for performing concurrent transcoding processes for a case of live transcoding and GOPs of differing transcoding time intervals1010-1017: Transcoding tasks assigned to integrated workers410of indices 0 to 71200: Delay of transcoded GOP stream with respect to a respective incoming GOP stream for a case of using integrated workers1300: Delay of transcoded GOP stream with respect to a respective incoming GOP stream for a case of using specialized workers1400: Resources of a large-scale transcoding station employing pools of workers1410: Compressed stream from source1420: Pool of type-1 workers (stage-1 workers)14211421: A type-1 worker1430: Reproduced source raw signal1440: Pool of type-2 workers (stage-2 workers)14411441: A type-2 worker1450: Processed reproduced source raw signal1460: Pool of type-3 workers (stage-3 workers)14611461: A type-3 worker1470: Compressed transcoded stream1500: Large-scale transcoding station employing pools of workers where at least one pool comprises clusters of workers of distinct characteristics1560: Cluster of workers;1560(j), 0≤j<(μ1−1), of stage-11570: Cluster of workers;1570(j), 0≤j<(μ2−1), of stage-21580: Cluster of workers;1580(j), 0≤j<(μ3−1), of stage-31600: Schematic of a large-scale transcoding station indicating heterogenous workers of stage-11610: Module for acquisition of a multimedia stream from a source140under control of an orchestrator16201620: An orchestrator controlling operations of the transcoding station1630: Module for communicating transcoded multimedia stream to a client under control of orchestrator16201640: Equipment for stage-1 (decoding stage) of the transcoding station1642: A module for directing individual GOPs to appropriate decoders1643: 1:N1selector, N1>11644: A worker (decoder) of a specific type1700: Schematic of the large-scale transcoding station indicating heterogenous workers of stage-21752: A module for directing individual GOPs to appropriate video-processing units (VPUs)1753: 1:N2selector, N2>11754: A worker (VPU) of a specific type1800: Schematic of the large-scale transcoding station indicating heterogenous workers of stage-31862: A module for directing individual GOPs to appropriate encoders1863: 1:N3selector, N3>11864: A worker (encoder) of a specific type1900: Forms of signals formed at a source directed to a transcoding station1910: A video stream arranged in groups of pictures (GOPs), also referenced as a group of frames1920: Compressed video stream of variable flow rate but equal compressed GOP durations1930: Compressed video stream of fixed flow rate, hence variable compressed GOP durations2000: Allocation of successive GOPs of video-signal1910to workers of the three stages2010: GOPs received during successive equal intervals2020: Indices of GOPs20102030: Duration of processing of a GOP2010within a first stage2035: Delayed processing2040: Duration of processing of a GOP within the second stage2050: Duration of processing of a GOP within the third stage2100: Allocation of successive GOPs of video-signal1920to workers of the three stages2110: GOPs received during successive variable intervals2120: Indices of GOPs21102130: Duration of processing of a GOP2110within a first stage2135: Delayed processing2140: Duration of processing of a GOP within the second stage2150: Duration of processing of a GOP within the third stage2200: Effect of varying durations of GOP processing on sequential order of processed GOPs2210: Instants of receiving GOPs2220: Indices of received GOPs2230: Sequential order of output GOPs of stage-12240: Sequential order of output GOPs of stage-22250: Sequential order of output GOPs of stage-32260: Process of rearranging the encoded GOPs of stage-3 according to corresponding temporal order at source2300: A transcoding station handing GOPs of a multimedia stream2310: Dual link connecting the transcoding station to network1202320: Network interface of the transcoding station2330: External two-way control data between the orchestrator and source equipment exchanged through network interface23202340: Upstream content data received from a source through the network interface23202342: Downstream data transmitted to a client (destination) through the network interface23202350: Orchestrator assembly comprising an orchestrator core and three stage controllers (detailed inFIG.24)2360: Internal control data exchanged between workers and the orchestrator assembly23502370: Content processing and transfer assembly (detailed inFIG.26andFIG.27)2400: Architecture of an orchestrator assembly where an orchestrator core communicates with workers of each stage through a respective stage controller2410: Transmission medium from a source (part of dual link2310)2420: Transmission medium to a sink (part of dual link2310)2450: Orchestrator core2451: Stage-1 controller2452: Stage-2 controller2453: Stage-3 controller2500: A stage controller2510: An interface with a pool of workers of a respective stage2520: Buffer holding identifiers of GOPs received from a previous stage2530: Buffer storing indices of available (free) workers2540: Stage-orchestrator interface2542: Dual channel carrying control data from orchestrator2543: Metadata of a new GOP2544: Identifier of a processed GOP2560: Processor (or an assembly of processors)2570: Stage scheduler (a software module)2600: Content-data flow within the transcoding station2610: Stage-1 content-transfer unit2620: Stage-2 content-transfer unit2630: Stage-3 content-transfer unit2631: Type-1 worker (stage-1 worker)2632: Type-2 worker (stage-2 worker)2633: Type-3 worker (stage-3 worker)2700: Overview of the transcoding station indicating connection of the orchestrator assembly to the content-processing assembly2710: Content-access unit between the network interface and the pool of stage-1 workers2720: Content-access unit between the pool of stage-1 workers and the pool of stage-2 workers2730: Content-access unit between the pool of stage-2 workers and the pool of stage-3 workers2740: Content-access unit between the pool of stage-3 workers and the network interface2750: Content-handling mechanism (an implementation of content processing and transfer assembly2370)2780: Compressed transcoded stream directed to at least one client180through the network interface23202790: Dual channel connecting the orchestrator core to content-access unit27402800: A mechanism for content transfer through the transcoding station2821: Channels connecting output port of switch-1, implemented as a 1:n selector (with n=6) to individual stage-1 workers (decoders)2822: Channels from individual stage-1 workers to input ports of switch-22823: Channels from output ports of switch-2 to stage-2 workers (video-processing units, VPUs)2824: Channels from stage-2 workers to input ports of switch-32825: Channels from output ports of switch-3 to stage-3 workers (encoders)2826: Channels from stage-3 workers to input ports of switch-4 (implemented as an m:1 selector, m=7)2840: 7:1 selector2870: Collator of decoded GOPs2900: A first example of a switching component of the mechanism for content access2910: n:1 Selector (n=6)2920: A single channel from n:1 selector2910to 1:y selector2930, y>12930: 1:y selector (y=4)3000: A second example of a switching component of the mechanism for content access3010: 12:1 Selector3011: 12:1 Selector3020: A single channel from selector3010to selector30303021: A single channel from selector3010to selector30313030: 1:4 selector3031: 1:4 selector3040: A pool of twelve workers30413041: A single worker of pool30403042: A single worker of pool30403060: A pool of seven workers3061: A single worker of pool30603062: A single worker of pool30603100: Processes performed at network interface2320comprising processes3110to31703200: Processes performed at the orchestrator assembly2350comprising processes3210to32603300: Further processes performed at the orchestrator assembly2350comprising processes3310to33403400: Processes performed at a stage controller, according to a first scheme of stage-specific worker allocation, comprising processes3410to34903500: Processes performed at a stage scheduler2570comprising processes3510to35903600: An example of worker selection implemented according to the first scheme of stage-specific worker allocation3610: Initial state of a circular buffer holding workers' identifiers3612: Worker identifier3614: Index of the circular buffer3620: An intermediate state of the circular buffer3622: Identifier of an available worker3630: Another intermediate state of the circular buffer3632: Identifier of an available worker3700: Steps of worker selection2800: Continued steps of worker selection3900: Workers organization into multiple task-specific worker bands3910: Number of workers of a specific stage, a specific task-type, and a specific form; a first example3920: Number of workers of a specific stage, a specific task-type, and a specific form; a second example4000: Second scheme of worker characterization4010: Task index4020: Maximum number of concurrent tasks4030: Position-identified tasks4100: Order of selection of available workers4110: Uni-functional workers4120: Multifunctional uni-tasking workers4130: Multifunctional multitasking workers4200: Mixture of uni-functional and multifunctional workers first example4210: Worker type4220: Number of workers4230: Task indices4240: Individual worker identifiers4300: Mixture of uni-functional and multifunctional workers second example4400: Sorted worker types4500: A data structure for facilitating worker selection in a transcoding station employing a mixture of uni-functional and multifunctional uni-tasking workers4550: The symbol “*” indicates a worker type that has not yet been provisioned4600: A data structure for facilitating worker selection in a transcoding station employing a mixture of uni-functional and multifunctional workers of both uni-taking and multitasking capabilities (data under the fields of “available resources” and “number of task-specific available workers” correspond to an initial state)4640: Number of available workers of a specific worker type configured to perform a specific task—for example,4640A indicates a number of available workers of work-type 4 configured to perform task 1,4640B indicates a number of available workers of work-type 6 configured to perform task 3, and4640C indicates a number of available workers of work-type 11 configured to perform task 14700: Exemplary entries of data structure4600in operation4800: Processes of worker selection of heterogenous sources comprising processes4810to48804900: Examples of uni-functional and multi-functional worker types5000: Examples of worker selection from a respective worker group of a same worker type (work type 14) for a case of 5 workers each configured to perform any of 14 tasks (versatility=14) but with a multi-tasking limit of 2 (at most two concurrent tasks per worker)5010: A first scenario of a fully occupied work group of worker-type 145020: A second scenario of a fully occupied work group of worker-type 145030: A third scenario of a fully occupied work group of worker-type 145040: A fourth scenario of a fully occupied work group of worker-type 145100: A procedure of constructing a data structure for facilitating allocation of workers comprising processes5110to51705200: A procedure, comprising processes5210to5290, for allocating tasks to workers for a case of a transcoding station employing a mixture of uni-functional and multifunctional workers where all workers are uni-tasking workers5300: A procedure, comprising processes5230to5390, for allocating tasks to workers for a case of a transcoding station employing a mixture of uni-functional and multifunctional workers with workers of both uni-tasking and multicasting capabilities5400: A transcoding station configured to concurrently transcode multiple streams using a shared orchestrator core5420: Multiple dual links connecting the network interface to a network5421: Overall control data from all connecting sources5422: Overall content data from all connecting sources and content data to destination sinks5430: An orchestrator assembly with a single orchestrator core configured to handle multiple streams5440: Dual control paths between orchestrator assembly5430and workers of the three stages5460: Overall stage-1 workers handling all streams5470: Overall stage-2 workers handling all streams6480: Overall stage-2 workers handling all streams5500: A transcoding station configured to concurrently transcode multiple streams using multiple orchestrator assemblies5530: A single orchestrator assembly5540: A set of orchestrator assemblies55305550: Dual control paths between the set of orchestrator assemblies and workers5600: A set of K orchestrator assemblies5530, K>1, with orchestrator assembly5530(j), 1≤j≤K, comprising a respective orchestrator core2450(j), a stage-1 controller5651(j), a stage-2 controller5652(j), and a stage-3 controller5653(j)5610: Dual channels carrying multimedia streams from respective sources and to respective sinks5650: An orchestrator assembly for a specific multimedia stream5651: Stage-1 controller of a respective orchestrator assembly5652: Stage-2 controller of a respective orchestrator assembly5653: Stage-3 controller of a respective orchestrator assembly5700: Multiple-stream resource sharing5760: Shared stage-1 workers including floating workers to be adaptively allocated to any stream as the need arises5770: Shared stage-2 workers including floating workers5780: Shared stage-3 workers including floating workers5800: Alternative worker-allocation policies5810: First worker-allocation policy based on segmentation of workers into stream-specific groups with an additional group of floating workers5811: Group of workers reserved for a first stream5812: Group of workers reserved for a second stream5813: Group of workers reserved for a third stream5814: Group of workers designated as floating workers5820: Second worker-allocation policy based on designating each worker as a floating worker5840: Workers adaptively allocated to streams5850: Individual workers5860: A pool of workers of a specific worker type allocated to handle GOPs of different streams5900: Overview of a large-scale transcoding station concurrently receiving multiple incoming multimedia streams and producing multiple transcoded streams per incoming stream according to different standards5920: Large-scale network interface6000: Procedure for measuring tasks' execution durations using training data and operational data comprising processes6010to60406100: Stage-1 measurements corresponding to specific stage-1 metadata6200: Stage-2 measurements corresponding to specific stage-2 metadata6300: Stage-3 measurements corresponding to specific stage-3 metadata

Notation

GOP-kT: A transcoded GOP of index k, k>0TG: Duration of a raw GOP as produced at sourceTT: Duration of a transcoding process using an integrated worker410TS1: Duration of performing a stage-1 function using a type-1 worker421TS2: Duration of performing a stage-2 task using a type-2 worker422Ts3: Duration of performing a type-3 task using a type-3 worker423TT+: A representative value of TTbased on statistical distribution of per GOP transcoding-process duration.Tsj+: j=1, 2, or 3, a representative value of TSj, determined from learned statistical distribution of per GOP specialized-processes duration.WT: Total number of integrated workersWS1: Number of type-1 workersWS2: Number of type-2 workersWS3: Number of type-3 workers

DETAILED DESCRIPTION

FIG.1illustrates a system100providing transcoding services comprising a network120supporting multimedia sources140, multiple transcoding stations160, two of which are illustrated, and clients180of the transcoding system.

A conventional transcoding server is configured to perform processes of:(1) acquiring baseband signals from modulated carriers received from multimedia sources;(2) detecting a transcoding request from a received signal;(3) extracting a stream of compressed group-of-pictures (GOPs) from an acquired baseband signal;(4) transcoding the stream of compressed GOPs to produce a stream of compressed transcoded GOPs; and(5) modulating a carrier signal with the output stream of compressed transcoded GOPs for transmission to at least one destination.

Generally, transcoding is performed in basic processing stages categorized as a decoding (decompression) stage, a signal-processing stage, and an encoding (compressing) stage, hereinafter referenced as the first stage, the second stage, and the third stage (stage-1, stage-2, stage-3), respectively. A compressed input stream of a specific raw multimedia stream is received from a multimedia source and supplied to a decoding stage which produces a decompressed stream approximately reproducing the specific raw multimedia stream. The decompressed stream is supplied to a signal-processing stage which performs conventional processes such as filtering and de-warping based on metadata embedded within the received transcoding request. The signal-processing stage produces a processed stream which is adapted to characteristics of a target client. The encoding stage compresses the processed stream to produce a compressed output stream according to a compression standard compatible with capabilities of the target client.

FIG.2is an overview200of a structure of a transcoding station160. A network interface220(detailed inFIG.3) connects to network120through a dual link210for receiving transcoding requests from the plurality of multimedia sources140and signal streams to be transcoded then communicating transcoded streams to respective destinations through the network120. The network interface220is configurated to separate control data (including metadata)230and content data240from each stream received from a multimedia source140. The network interface directs the control data230to an orchestrator250, through stream-specific control-data selector235, and the content data240to a pool260of shared workers of different types through high-capacity stream-specific content selector245. The orchestrator250receives stream-specific upstream control data231and transmits stream-specific downstream control data232to respective destinations through the network interface. The pool260of workers receives stream-specific upstream content data241and transmits stream-specific downstream content data242to respective destinations through the network interface220.

The orchestrator communicates with the pool260of workers through dual control paths270. A pool280of resources is communicatively coupled to the pool of workers260. A worker may be coupled to respective dedicated resources or rely on a pool of shared resources.

To enable concurrent transcoding at the transcoding stations160, the multimedia sources140generally format video-signal components of multimedia signals into groups of pictures (GOPs). A GOP comprises a number γ of video frames, γ>1 (γ=50, for examples) which may be source dependent and may even differ for GOPs of a same stream. At a frame rate of f frames per second, the duration of a GOP before compression at source is τ=(γ/f), and the GOP generation rate, denoted λ, is λ=(f/γ).

FIG.3illustrates components300of a network interface220. A source interaction module310receives transcoding requests from different multimedia sources140, communicates respective admission, or otherwise, to respective sources, and receives content data and corresponding control data and metadata from admitted sources. A Metadata extraction module320separates metadata and corresponding control data for a received stream to be directed to orchestrator250(FIG.2). A destination interaction module330communicates notifications and respective transcoded content to all destinations of each transcoded stream. A source-characterization algorithm340determines characteristics of each engaged multimedia source140, with the help of a source-characteristics database350. A content-data-extraction module360separates content (the payload, the data to be transcoded) of each stream to be directed to the pool260of workers (FIG.2). A module380for rearranging transcoded GOPs collates GOPs according to streams and orders transcoded GOPs of each stream to correspond to the temporal order of corresponding incoming GOPs. A network-interface processor (or a pool of processors)390executes software instructions of modules310,320,330,340,360, and380.

As described above, a transcoding process is implemented in three stages. In a first stage, a received GOP is decompressed. In a second stage, conventional signal processing may be performed to condition the individual frames of the decompressed GOP according to respective metadata. In a third stage, the processed GOPs are encoded according to a standard specified in the respective meta data.

FIG.4provides a clarification400of using integrated workers versus using specialized workers. The pool of workers260may comprise integrated workers410and/or sets420of specialized workers. An integrated worker410sequentially performs processes of decompression, signal processing, and encoding of the aforementioned three stages. A set420of specialized workers comprises:a specialized worker421performing decompression of a compressed multimedia signal of a specified standard (referenced as a type-1 worker or a stage-1 worker)a specialised worker422performing conventional signal processes (referenced as a type-2 worker or a stage-2 worker); anda specialized worker423performing multimedia encoding, including compression, according to a requisite standard (referenced as a type-3 worker or a stage-3 worker).

Live Multimedia Transcoding Versus Multimedia-File Transcoding

Structuring a multimedia stream into independent GOPs enables concurrent transcoding of multiple GOPs. Concurrent transcoding may be performed using a group of integrated workers410or groups of specialized workers comprising a first group of type-1 workers421, a second group of type-2 workers422, and a third group of type-33 workers423.

For a task of transcoding a recorded multimedia session, where encoded data of an entire multimedia session is stored in an accessible memory device, the number of workers of any type may vary from one (hence processing one GOP at a time) to an arbitrary number. The period of time taken to complete transcoding an entire stored multimedia session decreases as the number of workers increases. Using integrated workers, the number of workers which may be employed concurrently varies from one to a total number of GOPs of an entire session. Using specialized workers, the number of workers of the most processing-intensive type may vary from one to the total number of GOPs of the session. Thus, the provisioning of workers is based on a trade off between overall transcoding time and cost.

For a task of live transcoding of a multimedia stream, the number of workers of any type exceeds one and is determined according known task-execution durations as illustrated inFIG.5. For example, if an appropriate number of integrated workers410is determined to have a lower bound of 20 and an upper bound of 24, then live transcoding cannot be realized with less than 20 integrated workers and employing more than 24 integrated workers cannot expedite the overall transcoding task since an idle worker cannot process a forthcoming GOP.

FIG.5provides a comparison500of durations of transcoding functions using integrated workers410versus using sets420of specialized workers421,422, and423. The duration of a raw GOP (as produced at source) is denoted TG. The duration of a transcoding process using an integrated worker is denoted TT. The duration TGmay vary from one GOP to another, even under the same frame rate, if the number of frames per GOP varies within a same stream. The duration TTmay differ from one GOP to another, even with a constant number of frames per GOP, depending on the contents of the GOPs.

For a simplified case where both TGand TTare invariant for successive GOPs, the number of integrated workers410, denoted WT, needed to eliminate the need to store received GOPs within a transcoding station200is determined as WT≥┌TT/TG┐. With a duration TTvarying for successive GOPs, the requisite number of integrated workers would be: WT≥┌TT+/TG┐, where TT+is determined from learned statistical distribution of per GOP transcoding-process duration.

For the simplified case where both TGand TTare invariant for successive GOPs, the number of specialized workers needed to eliminate the need to store received GOPs within the transcoding station200is determined as follows:the number of type-1 workers, denoted WS1, is determined as WS1≥┌TS1/TG┐;the number of type-2 workers, denoted WS2, is determined as WS2≥┌TS2/TG┐; andthe number of type-3 workers, denoted WS3, is determined as WS3≥┌TS3/TG┐,
where TSjdenotes the duration of processes performed in stage-j, j=1, 2, or 3.

With a duration TSj, j=1, 2, or 3, varying for successive GOPs, the requisite number of specialized workers would be WSj≥┌TSj+/TG┐, where TSj+is determined from learned statistical distribution of per GOP specialized-processes duration.

FIG.6is an overview600of concurrent transcoding using integrated workers410. Process610transcodes received multimedia signals612organized into independent GOPs which are encoded (compressed) at source according to a first standard to be transcoded according to a second standard. The received GOPs are transcoded independently, according to metadata614defining structure of incoming GOPs, using a number WTof integrated workers410. Metadata614may identify the second standard according to which the transcoded output is to be formed. Process620collates the transcoded GOPs, in the temporal order in which corresponding incoming GOPs were received, to produce a sequence of transcoded GOPS622arranged in proper order.

FIG.7illustrates a scheme700for concurrent transcoding of GOPs using specialized workers240. Process712uses type-1 workers421to decode (decompress) received multimedia signals612organized into independent GOPs which are encoded (compressed) at source according to a first standard to be transcoded according to a second standard. Process714performs conventional signal processing of decoded GOPs, received from type-1 workers, using type-2 workers422. Process716encodes (compresses) processed GOPs, received from type-2 workers, using type-3 workers423. Process720collates transcoded GOPs, received from type-3 workers423, to produce a stream of transcoded GOPs of the same sequential order of corresponding incoming GOPs of multimedia stream612. The metadata614is supplied to processes712,714, and716.

FIG.8is a representation800of processing time intervals of independently transcoding GOPs to produce transcoded GOPs {GOP-0T, GOP-1T, etc.} for a case of GOPs of equal processing time intervals. Successive GOPs received at transcoding station200are identified according to indices810. Using an appropriate number of integrated workers410, for each incoming GOP transcoding starts immediately after completion of acquisition of the GOP and ends after a time interval820. Thus, realizing live transcoding where an outgoing transcoded stream is transmitted after a relatively short period of time following arrival of a respective incoming stream.

FIG.9is a representation900of processing time intervals of independently transcoding GOPs to produce transcoded GOPs {GOP-0T, GOP-1T, etc.} for a case where the processing time required to transcode a GOP may differ significantly from one GOP to another. Using an appropriate number of integrated workers410, for each incoming GOP transcoding starts immediately after completion of acquisition of the GOP and ends after a time interval920. Thus, realizing live transcoding where an outgoing transcoded stream is transmitted after a relatively short period of time following arrival of a respective incoming stream.

FIG.10illustrates assignment of GOPs to a number of integrated workers for performing concurrent transcoding processes for the case of GOPs of equal transcoding time intervals (illustrated inFIG.8). Transcoding tasks assigned to each worker of index j, 0≤j<7, are identified (reference numerals1010to1016).

FIG.11illustrates assignment of GOPs to a number of integrated workers for performing concurrent transcoding processes for the case of GOPs of differing transcoding time intervals (illustrated inFIG.9). Transcoding tasks assigned to each worker of index j, 0≤j<8, are identified (reference numerals1110to1117).

FIG.12illustrates an example1200of the delay1240incurred in transcoding an incoming GOP stream for a case of using integrated workers410.

FIG.13illustrates an example1300of the delay1340incurred in transcoding an incoming GOP stream for a case of using specialized workers420.

FIG.14illustrates resources1400provided at a large-scale transcoding station comprising a first stage comprising a pool1420of decoding workers1421, a second stage comprising a pool1440of signal-processing workers1441, and a third stage comprising a pool1460of encoding workers1461. The signal-processing workers mainly perform video-signal operations and are hereinafter referenced as video-processing units (VPUs).

The transcoding station receives a stream1410of compressed GOPs from a source140.

Each of the corresponding raw GOPs at the source comprises a known number of video frames with a known rate of frames per second. The raw GOPs are compressed at source according to any of standardized methods to produce compressed GOPs. The sizes (number of bytes) of compressed GOPs may vary significantly according to intra-frame and inter-frame view variation. The durations of the compressed GOPs may be equal, if the stream is transmitted from the source at a variable bit rate, or time-variant if the stream is transmitted at a constant bit rate as illustrated inFIG.19.

The output stream1430of stage-1 is an approximation of the source raw stream. Regardless of the durations of the compressed GOPs of stream1410, the decoding time intervals (workers' engagement periods) in stage-1 may vary significantly and may considerably exceed the mean-value τ0of durations of the received compressed GOPs. To circumvent the need for massive storage of compressed GOPs at input of the first stage, multiple stage-1 decoders1421are employed so that several compressed GOPs may undergo decoding processes concurrently in different decoders1421. Due to the variation of decoding time intervals of different GOPs, the output stream of stage-1 may comprise decoded GOPs produced in an order that differs from the order of respective compressed GOPs as illustrated inFIG.20andFIG.22. For a steady-state operation, the collective decoding capability of stage-1 is selected so that the mean value τ1of the completion time intervals of successive decoded GOPs does not exceed τ0.

The pool1440of video-processing units (VPUs)1441processes stream1430of decoded GOPs where several decoded GOPs may be processed concurrently in different VPUs.

Due to the variation of processing time intervals of different decoded GOPs of stream1430, the output stream1450of stage-2 may comprise decoded GOPs produced in an order that differs from the order of respective decoded GOPs of stream1430as illustrated inFIG.20andFIG.22. For a steady-state operation, the collective signal-processing capability of stage-2 is selected so that the mean value τ2of the completion time intervals of successive processed GOPs within stage-2 does not exceed τ0; this is an essential design requirement.

The output stream1450of stage-2 comprises processed GOPs compatible with capabilities of intended receiving clients180(the transcoded stream may be directed to multiple clients180of similar characteristics). The pool1460of encoding units1461encodes stream1450of processed GOPs where several processed GOPs may be encoded concurrently in different encoders1461. Due to the variation of encoding time intervals of different processed GOPs, the output stream of stage-3 may comprise decoded GOPs produced in an order that differs from the order of respective processed GOPs of stream1450as illustrated inFIG.20andFIG.22. For a steady-state operation, the collective encoding capability of stage-3 is selected so that the mean value of the completion time intervals of successive encoded GOPs within stage-3 does not exceed τ0.

The output stream1470of stage-3 comprises compressed encoded GOPs compatible with capabilities of intended receiving clients140. The GOPs of the output stream1470over a moving time window may need to be buffered to enable re-ordering of the GOPs as needed.

The mean values τ1, τ2, and τ3of the completion time intervals, together with corresponding standard deviations σ1, σ2, and σ3, may be determined from measurements to be used for optimal provisioning of resources of the transcoding station.

FIG.15illustrates details1500of the transcoding-station resources ofFIG.14where at least one pool of workers is arranged into multiple clusters of workers of distinct characteristics.

In general, the pool1420of decoding workers comprises μ1clusters, μ1≥1, referenced as1560(0) to1560(μ1−1) of decoding workers of distinct forms, the pool1440of signal processing workers comprises μ2clusters, μ2≥1, referenced as1570(0) to1570(μ2−1) of distinct signal-processing workers, and the pool1460of encoders comprises μ3clusters, μ3−1, referenced as1580(0) to1580(μ3−1), of distinct encoders.

With a focus on selected clusters of resources applicable to a specific stream, a cluster1560of N1decoders, N11, a cluster1570of N2signal adaptors, N2≥1, and a cluster1580of N3encoders, N3≥1, are used inFIG.16,FIG.17, andFIG.18.

FIG.16is a schematic1600illustrating a large-scale transcoding apparatus comprising: an orchestrator1620of a transcoding station directing operation of: a module1610configured to perform a process of acquisition of a multimedia stream from a multimedia source140; a decoding stage1640; a signal-processing stage1650, an encoding stage1660; and a module1630for communicating transcoded multimedia stream to a client180under control of orchestrator1620. A module1642is configured to direct a stream of GOPs to an appropriate decoder cluster1560, through a 1:N1selector, and direct individual GOPs of the stream to available decoders1644within the cluster.

FIG.17is schematic1700illustrating the large-scale transcoding station ofFIG.16detailing processes of the signal-processing stage1650. A module1752is configured to direct a stream of decoded GOPs to an appropriate VPU cluster1570, through a 1:N2selector, and direct individual decoded GOPs of the stream to available VPUs1754within the cluster.

FIG.18is a schematic1800illustrating the large-scale transcoding station ofFIG.16andFIG.17detailing processes of the encoding stage1860. A module1862is configured to direct a stream of processed GOPs to an appropriate encoder cluster1580, through a 1:N3selector, and direct individual processed GOPs of the stream to available encoders1864within the cluster.

FIG.19illustrates video-stream components1900of multimedia streams formed at a multimedia source140to be directed to a transcoding station160. The video streams may comprise variable-flow-rate streams with fixed durations of group-of-pictures (GOPs), or fixed-flow-rate streams with variable GOP durations. A raw video stream1910originating at a multimedia source140is arranged in raw groups of pictures (GOPs). Six raw GOPs, indexed as 000 to 005, are illustrated. The raw video stream1910may be compressed to form a compressed video stream1920of a variable flow rate but equal compressed GOP durations, or a compressed video stream1930of a fixed flow rate, hence variable compressed GOP durations.

FIG.20illustrates allocation2000of successive GOPs of video-stream1920, of a variable flow rate, constant inter-GOP periods, to workers of the three stages ofFIG.15

Within the first stage, compressed GOPs2010, of indices2020, are received at successive time instants tj, where the intervals (tj+1-tj), j≥0, are equal. The durations2030of processing individual GOPs2010within the first stage may vary significantly; consequently, the temporal order of generating decoded GOPs may differ from the order of receiving the GOPs from the source. With the illustrated successive GOPs denoted GOP-0to GOP-9, and the processing durations of individual GOPs indicated with respective thick line spans, it is seen that the decoded GOPs, labeled [GOP-1] to [GOP-8], corresponding to the incoming compressed GOP-0to GOP-8, are produced in the order [GOP-1], [GOP-0], [GOP-2], [GOP-4], [GOP-3], [GOP-5], [GOP-6], [GOP-7], [GOP-8].

Within the second-stage, decoded GOPs derived from received GOPs2010, are received during generally unequal intervals. The duration2040of processing of a GOP within the second stage may also vary; consequently, the temporal order of generating processed decoded GOPs may differ from the temporal order at which the decoded GOPs are received from the first stage. As indicated (reference2035), when decoding of the GOP of index 2 in stage 1 is completed, neither of the two workers of stage 2 is available, hence the decoded GOP is queued at worker(0) of stage-2 for a short interval before processing.

The processed GOPs, labeled [[GOP-0]] to [[GOP-8]], corresponding to the decoded [GOP-0] to [GOP-8], are produced in the order [[GOP-1]]. [[GOP-2]], [[GOP-0]], [[GOP-4]], [[GOP-3]], [GOP-5]], . . .

Within the third stage, the processed decoded GOPs derived from decoded GOPs, are also received during generally unequal intervals. The duration2050of processing of a GOP within the third stage may vary significantly, according to standard to which the stream is encoded.

FIG.21illustrates allocation2100of successive GOPs of video-stream1930, of a fixed flow rate, variable inter-GOP periods, to workers of the three stages ofFIG.15.

Within the first stage, compressed GOPs2110, of indices2120, are received at successive intervals tj, where the intervals (tj+1-tj), j≥0, are generally unequal. The durations2130of processing individual GOPs2110within the first stage may vary significantly; consequently, the temporal order of generating decoded GOPs may differ from the order of receiving the GOPs from the source.

Within the second stage, decoded GOPs derived from received GOPs2110, are received during generally unequal intervals. The duration2140of processing of a GOP within the second stage may also vary; consequently, the temporal order of generating processed decoded GOPs may differ from the temporal order at which the decoded GOPs are received from the first stage. As indicated (reference2135), when decoding of the GOP of index 2 in stage 1 is completed, neither of the two workers of stage 2 is available, hence the decoded GOP is queued at worker of index 0 of stage-2 for a short interval before processing.

Within the third stage, the processed decoded GOPs derived from decoded GOPs, are also received during generally unequal intervals. The duration2150of processing of a GOP within the third stage may vary significantly, according to protocol to which the stream is encoded.

FIG.22illustrates an example2200of discrepancy between the order of arrival of received GOPs and the order of producing transcoded GOPs due to varying processing time intervals of different GOPs within each of the three stages. The GOPs received at successive time instants2210are indexed sequentially, in steps of 1 (reference2220). Received GOPs of indices 0 to 15 are illustrated. As illustrated, the sequential order of the decoded GOPs2230in stage-1 differs from the sequential order of the GOPs received from the source. The sequential order of the processed GOPs2240in the second stage differs from the sequential order of decoded GOPs2230. The sequential order of the encoded GOPs2250in the third stage differs from the sequential order of processed GOPs2240.

A process2260rearranges the encoded GOPs of stage-3 in the same temporal order of corresponding raw GOPs formed at the source, prior to transmission to a destination client180. Rearrangement of the encoded GOPs may be performed at the network interface220/2320or at a stage controller (an arbitrator) associated with the orchestrator250. To enable the rearrangement, a number of encoded GOPs, within a moving time window, may need to be held in a circular buffer.

FIG.23is an overview2300of a transcoding station, handling GOPs of a single multimedia signal, comprising an orchestrator assembly2350and a content-transfer assembly2370configured to transfer GOPs content to the pools of workers illustrated inFIG.14.

Network interface2320communicates with multimedia sources140and clients180through a dual link2310connecting the transcoding station to network120. The orchestrator assembly2350exchanges external control data2330with multimedia sources140and clients180. The orchestrator assembly2350distributes internal control data2360to the content-transfer assembly.

Under control of the orchestrator assembly2350, the content-transfer assembly2370receives upstream content data2340from a multimedia source140and transmits downstream data2342to a client180through the network interface2320.

FIG.24illustrates an architecture2400of the orchestrator assembly2350, where an orchestrator core2450communicates with workers of each stage through a respective stage controller. Network interface2320receives compressed streams from multimedia sources240through a transmission medium2410from network120and transmits transcoded compressed streams to clients180through a transmission medium2420to network120(dual link2310constitute transmission medium2410and transmission medium2420).

FIG.25illustrates a structure2500of a stage controller2550(one of2451,2452, and2453) comprising:an interface2510with a cluster of workers (FIG.15) of a respective stage;an interface2540with the orchestrator core2450;a buffer2520holding identifiers of GOPs received from a previous stage;a buffer2530storing indices of released workers;a memory device2570storing a scheduler module (software instructions); anda processor (or an assembly of processors)2560coupled to interface2510, interface2540, buffer2520, buffer2530, and memory device2570.

A dual channel2542from/to orchestrator core2450carries metadata2543of a new GOP to be scheduled, and an identifier2544of a successfully scheduled GOP.

Content-transfer unit2610transfers stream1410of compressed GOPs, received from a source240through network interface2320, to a selected cluster of stage-1 workers for decoding and transfers the decoded stream1430of GOPs to content-transfer unit2620.

Content-transfer unit2620transfers the decoded stream1430to a selected cluster of stage-2 workers for performing selected signal-processing operations, as indicated in respective metadata, and transfers the processed stream1440to content-transfer unit2630.

Content-transfer unit2630transfers the decoded stream1430to a selected cluster of stage-3 workers for encoding to a specific standard, according to orchestrator-core instructions, and transfers the encoded (compressed) stream1470to the network interface2320to be delivered through network120to a specified client180, or a designated set of clients180.

Heterogenous Resources

The first content-access unit directs a stream1410of compressed GOPs from a source240to a specific stage-1 cluster selected at stage-1 controller2451. The second content-access unit directs reproduced source raw signal1430to a specific stage-2 cluster selected at stage-2 controller2452. The third content-access unit directs processed reproduced source raw signal to a specific stage-3 cluster1460selected at stage-3 controller2453. The fourth content-access unit, communicatively coupled to the orchestrator core through dual channel2790, directs compressed transcoded stream2780to the network interface2320for transmission to at least one client180.

FIG.28illustrates an exemplary implementation2800of the content-processing assembly2750. The selected stage-1 cluster comprises six workers (six decoders). The selected stage-2 cluster comprises four workers (four VPUs). The selected stage-3 cluster comprises seven workers (seven encoders). Content-access units2710,2720, and2730are implemented as a 1:6 selector, a 6:4 switching unit, and a 4:7 switching unit respectively. Content-access unit2740is implemented as a 7:1 selector2840and a collator2870of decoded GOPs.

Channels2821connect output ports of switch-1, implemented as a 1:n selector (n=6, in the illustrated example) to individual stage-1 workers (decoders). Channels2822connects individual stage-1 workers to input ports of switch-2 having 6 input ports and 4 output ports. Channels2823connects output ports of switch-2 to stage-2 workers (video-processing units, VPUs). Channels2824connect stage-2 workers to input ports of switch-3 having 4 input ports and 7 output ports. Channels2825connect output ports of switch-3 to stage-3 workers (encoders). Channels2826connect stage-3 workers to input ports of switch-4 (implemented as an m:1 selector, m=7, in the illustrated example).

Switch-1, implemented as a 1:6 selector, directs individual GOPs of compressed stream1410to respective allocated decoders through channels2821as determined at a controller of switch-1 (not illustrated). Switch-2, implemented as a 6:4 switch, directs individual decoded GOPs to respective allocated VPUs through channels2823as determined at a controller of switch-2 (not illustrated). Switch-3, implemented as a 4:7 switch, directs individual processed GOPs to respective allocated encoders through channels2825as determined at a controller of switch-3 (not illustrated). Switch-4, implemented as a 7:1 selector, directs individual encoded GOPs to the network interface2320.

In a conventional m1:m2switch (m1input ports and m2output ports), m1>1, m2>1, all of the m1input ports and all of the m2 output ports may be active simultaneously. However, in the content-processing assembly ofFIG.28, an input port of any switch connecting at input to workers is only activated when a respective worker completes a respective function (decoding in stage-1, signal-processing in stage-2, or encoding in stage-3). With the processing time duration at any of the three stages substantially exceeding the transfer time of a GOP (hence the need for multiple workers for each stage), only a subset of input ports of a switch is activated simultaneously. Accordingly, the switching mechanisms can be simplified, in comparison with a switching mechanism of a conventional switch.

FIG.29illustrates an implementation2900of a 6:4 switch (m1=6, m2=4) comprising a 6:1 selector2910, a 1:4 selector2930, and a channel2920connecting the output port of selector2910to the input port of selector2930. Only one GOP may be transferred at a time. Thus, an output GOP of a worker of cluster1420may be buffered if another GOP is in transit along channel2920. As mentioned above, the transit time of a processed GOP is typically much smaller than the processing time at a respective worker.

FIG.30illustrates an implementation3000of a 12:8 switch (m1=12, m2=8) comprising a two 12:1 selectors,3010and3011, two 1:4 selectors,3030and3031, a channel3020connecting the output port of selector3010to the input port of selector3030, and a channel3021connecting the output port of selector3011to the input port of selector3031. Two GOP may be transferred concurrently from cluster3040of workers to cluster3060of workers.

FIG.31illustrates processes3100performed at the network interface2320. In process3110, the network-interface receives a transcoding request, and corresponding metadata, for transcoding a specific multimedia stream from a multimedia source140. The metadata includes information defining the standard according to which the video component of the specific multimedia stream is encoded, the requisite standard to which the transcoded stream is to be encoded, and other relevant characterization of the specific multimedia stream such as the size of a GOP and the video-component frame rate. In process3120, the network interface communicates the metadata to the orchestrator (1350,FIG.23,FIG.24). Upon receiving a response from the orchestrator, the network interface sends information regarding availability time to the multimedia source (process3130) then receives the multimedia stream from the source (process3140), assigns a GOP identifier to each GOP, and prefixes GOP identifiers to metadata and contents of respective GOPs (process3150) to enable tracking processed GOPs which may experience temporal scrambling within the transcoding station due to variation of GOPs' processing times. Preferably, the GOP identifiers are cyclical integers in steps of 1, starting with 0; such as 0 to 127, for example, since the transcoding station would not at any time be handling more than a relatively small number, 16 for example, of most recent GOPs. The network-interface sends the GOPs metadata marked with corresponding GOPs identifiers to stage-1 controller of the orchestrator assembly (process3160) and sends content data (process3170), marked with corresponding GOPs identifiers, to content transfer unit2610,FIG.26,2710,FIG.27.

FIG.32illustrates processes3200performed at the orchestrator assembly2350(FIG.23,FIG.24) to initialize stream-specific transcoding processes. In process3210, the orchestrator assembly receives a request and respective metadata from the multimedia source, through the interface as illustrated inFIG.31. The orchestrator core2450determines availability time instant based on current resource (workers) occupancy and communicates the availability time instant to the multimedia source through the network interface (process3220). Upon receiving confirmation from the multimedia source (process3230), the orchestrator core initializes schedulers of the three stages (process3240) and allocates workers to each stage based on the metadata (process3250). The three-stage transcoding functions are then performed for the duration of the multi-media stream (process3260).

FIG.33illustrates pipelined processes3300performed at the orchestrator assembly2350. In process3310, the orchestrator core continuously receives from each stage controller an identifier of a respective processed GOP and a count of waiting GOPs. The stage-1 workers, the stage-2 workers, and the stage-3 workers may concurrently be handling several GOPs and the orchestrator core is made aware of the progress in each of the three stages.

The orchestrator core transfers (process3320):an identifier of a decoded GOP in stage 1 to controller2452of stage 2;an identifier of a processed GOP in stage 2 to controller2453of stage-3; andan identifier of an encoded GOP in stage-3 to network interface2320through control-data path2330.

The content-transfer assembly concurrently directs (process3330):the decoded GOP in stage 1 (output of stage-1 encoders) to designated workers (VPUs) of stage-2;the processed GOP in stage-3 (output of stage-2 VPUs) to designated encoders of stage-3; andthe encoded GOP in stage-3 to network-interface2320through content-data path2780(FIG.27).

Subject to a determination that a count of waiting GOPs at any stage exceeds a respective predetermined threshold (permissible level), the orchestrator core instructs (process3340) the multimedia source to pause transmission of the multimedia stream to be resumed when the count of waiting GOPs is below the threshold. Thus, if the waiting GOPs in any of stage-1, stage-2, or stage-3 exceeding a respective permissible level, the source pauses transmission. Determination of the permissible levels takes into account the two-way transfer delay between the transcoding station and the multimedia source. In a network120configured to handle high-quality streaming services, the dual transfer delay would be a small fraction of a second between any two points on the planet. The duration of a typical GOP is of the order of one second; thus, even repetitive pausing and resuming transmission from the multi-media source would still allow smooth operation of the transcoding processes.

FIG.34illustrates processes3400performed at a stage-controller2500(2451,2452, or2453) according to a first scheme of worker allocation. Upon receiving (process3410) metadata of a new-GOP from the orchestrator core, a stage-orchestrator interface2540causes processor2560to queue the new-GOP metadata in buffer2520(process3420) and update a count of waiting GOPs (process3430). The stage scheduler2570is activated to cause processor2560to allocate a worker of the stage to a selected waiting GOP (process3440). The stage scheduler2570returns a state 0 if an appropriate worker is not available or a state “1” together with an identifier of an allocated worker (process3450). If an appropriate worker is not available, stage-orchestrator interface2540communicates (process3460) the count of waiting GOPs to the orchestrator core2450. Otherwise, the count of waiting GOPs is reduced (process3470), the content of the selected waiting GOP is transferred to the allocated worker through a content-transfer path (process3480), and an identifier of the selected GOP is communicated to the subsequent stage.

As illustrated inFIG.15, the transcoding station employs μ1distinct clusters of decoding workers, μ2distinct clusters of signal-processing workers, and μ3distinct clusters of encoders, μ1≥1, μ2≥1, μ3≥1. Each cluster of workers is specific to a stage and a worker type. The workers of stage-1, for example, may comprise a number of worker-clusters each for decoding a received compressed stream formed according to a specific standard (such as H.264, H.265, etc.). A scheduler2570is dedicated to a specific worker-type of a specific stage. According to the first scheme of stage-specific worker allocation, a cluster-specific number of workers of a same type is reserved for each cluster.

Identifiers of the workers of a cluster are placed in a circular buffer in any order at entries indexed as 0 to (ν−1), ν being a provisioned number of same-type workers of a cluster, ν>1. With W(j), 0≤j<ν, denoting a worker's identifier placed in entry j, an initial selection of W(j) is selected to equal j. A first index, denoted index1, points to an entry in the circular buffer holding an identifier of an available worker. A second index, denoted Index2, points to an entry in the circular buffer in which an identifier of a worker, of the cluster of workers, that has just completed a task relevant to a respective GOP is to be written. An integer β denotes a number of workers of the provisioned workers that are occupied at a given instant of time; 0≤β≤ν. Each of Index1, Index2, and β is initialized as integer zero.

FIG.35illustrates processes3500performed at a stage scheduler2570according to the first scheme of stage-specific worker allocation. Process3510initializes each of Index1, index2, and β to equal integer 0, setting W(j) to equal j, for 0≤j<ν. Process3520starts executing the processes of allocating a worker, from the cluster of workers, to process a GOP of an identifier queued in buffer2520, if any. In process3530, processer2560accesses buffer2530storing indices of released workers. If process3540determines that buffer2530is empty, process3560is activated. If process3540determines that a released worker is found, process3550is activated to return the released worker where an identifier of the released worker is written in entry Index2of the circular buffer, the number β of occupied workers is reduced to (β−1), and index2is updated to (index2+1)|ν(X modulo Y, where X and Y are positive integers is conventionally denoted X|Y). Buffer2520holding identifiers of GOPs to be processed is then accessed to read an identifier of a waiting GOP (process3560).

Process3565initializes as zero an indication of successful allocation of a worker to a waiting GOP. In process3570if buffer2520is found to be empty or the number of occupied workers has already reached the maximum value ν, process3530is revisited. Otherwise, If process3570determines that there is a waiting GOP and the number of occupied workers is less than ν, process3580declares that a worker is allocated to the waiting GOP and process3590selects worker W(index1), increases the number β of occupied workers to (β+1), and updates index1to (index1+1)|ν.

FIG.36illustrates an example3600of worker selection implemented according to the first scheme of stage-specific worker allocation for a case of ν=8. The initial state3610of a circular buffer holding worker identifiers 0 to 7 is illustrated with β=0, and Index1=index2=0, and worker indices (reference3612) set as W(j)=j, 0≤j<8, j being the index3614of the circular buffer. At an intermediate state3620of the circular buffer, following handling several GOPs, Index1=7, index2=3, β=4, and the order of available workers becomes scrambled, due to varying GOPs' processing durations; for example, worker identifier3622at index 0 is now 2 instead of 0 (W(0)=2). At a further intermediate state3630of the circular buffer, Index1=4, index2=7, β=5, and the order of available workers is scrambled; worker identifier3632at index 6 is 4 (W(6)=4), for example.

FIG.37is a walkthrough3700of the process ofFIG.36, illustrating steps of workers selection.FIG.38is a continuation3800ofFIG.37. The states of the circular buffer are identified sequentially; states (0) to (19) are illustrated.

In state (1), worker 0 is allocated to a GOP, J=1, K=0, β=1, and W(0)=null.

In state (3), worker 2 is allocated to another GOP, J=3, K=0, β=3, and W(j)=null, 0≤j<3.

In state (4), worker 3 is allocated to another GOP, J=4, K=0, β=4, and W(j)=null, 0≤j<4.

In state (5), worker 2 is released and placed in entry K=0, and K is increased to 1, hence J=4, K=1, β=3, and W(j)=null, 1≤j<4.

In state (6), worker 4 is allocated to a new GOP, J=5, K=1, β=4, and W(j)=null, 1≤j<5.

In state (7), worker 0 is released and placed in entry K=1, and K is increased to 2, hence J=5, K=2, β=3, and W(j)=null, 2≤j<5.

In state (8), worker 5 is allocated to a new GOP, J=6, K=2, β=4, and W(j)=null, 2≤j<6.

In state (9), worker 1 is released and placed in entry K=2, and K is increased to 3, hence J=6, K=3, β=3, and W(j)=null, 3j<6.

In state (10), worker 6 is allocated to a new GOP, J=7, K=3, β=4, and W(j)=null, 3≤j<7.

In state (11), worker 7 is allocated to a new GOP, J=8|8=0, K=3, β=5, and W(j)=null, 3≤j<8.

In state (12), worker 2 is allocated to a new GOP, J=1, K=3, β=6, and W(j)=null, 3≤j<8, and j=0.

In state (13), worker 0 is allocated to a new GOP, J=2, K=3, β=7, and W(j)=null, 3≤j<8, and j=0, 1.

In state (14), worker 1 is allocated to a new GOP, J=3, K=3, β=8, and W(j)=null, 3≤j<8, and j=0, 1, 2. J=K when all workers of the cluster are free (as initialized in state 0) or all workers of the cluster are occupied (state 14 in this example).

In state (15), worker 5 is released and placed in entry K=3, and K is increased to 4, hence J=3, K=4, β=7, and W(j)=null, 4≤j<8, and j=0, 1, 3.

In state (16), worker 3 is released and placed in entry K=4, and K is increased to 5, hence J=3, K=5, β=6, and W(j)=null, 5≤j<8, and j=0, 1, 3.

In state (17), worker 2 is released and placed in entry K=5, and K is increased to 6, hence J=3, K=6, β=5, and W(j)=null, j=6,7, and j=0, 1, 3.

In state (18), worker 5 is allocated to a new GOP, J=4, K=6, β=6, and W(j)=null, j=6, 7, and 0≤j<4.

In state (19), worker 4 is released and placed in entry K=6, and K is increased to 7, hence J=4, K=7, β=5, and W(j)=null, j=7, and 0≤j<4.

FIG.39illustrates workers organization3900into multiple task-specific worker bands3900, for any of the three stages. For the illustrated case, four task types, labeled task-type-0, to task-type-3, are applicable for a specific stage and workers configured to perform each task-type are sorted into a respective number of bands (four bands for task-type-0, five bands for task-type-1, four bands for task-type-2, and six bands for task-type-3. The first allocation scheme ofFIG.35is applied to a set of bands comprising a relevant band for each task type. The number of workers per band may vary significantly; for example, band-0 of task-type-2 contains 5 workers (reference3910) while band-3 of task-type-3 contains 11 workers.

As illustrated inFIGS.14,15,23,24, and27, the transcoding station160comprises a network interface2320, a pool of decoders1420, a pool of signal-adaptors1440, a pool of encoders1460, an orchestrator assembly2350, and a content-processing assembly2370,2750.

The network interface is configured to receive, from a multimedia source140, a transcoding request and a multimedia stream comprising a video stream1900organized into source groups-of-pictures (GOPs). The transcoding request indicates a first standard according to which the multimedia stream is formed and a second standard according to which an output decoded stream is to be formed.

The pool of decoders is configured to concurrently decode different GOPs, according to the first standard, to produce respective decoded GOPs. The pool of signal-adaptors is configured to concurrently process different decoded GOPs to produce respective adapted GOPs. The pool of encoders is configured to concurrently encode different adapted GOPs to produce respective encoded GOPs according to the second standard;

The orchestrator assembly2350is configured to assign each source GOP to an available decoder, each decoded GOP to an available signal adaptor, and each adapted GOP to an available encoder. The content-processing assembly2750is configured to provide each source GOP access to an available decoder, each decoded GOP access to an available signal adaptor, and each adapted GOP access to an available encoder. An output-collating module, coupled to the network interface, is configured to arrange successive encoded GOPs in the order of corresponding source GOPs.

Each decoder of the pool of decoders, each signal adaptor of the pool of signal adaptors, and each encoder of the pool of encoders comprises a respective hardware processor coupled to a memory device storing software instructions, and a buffer holding intermediate data.

The orchestrator assembly2350is further configured to determine, in response to the transcoding request, availability time based on current occupancy of the pool of decoders, the pool of signal adaptors, and the pool of encoders. The orchestrator assembly2350communicates the availability time to the multimedia source140through the network interface2320.

The orchestrator assembly2350is further configured to track counts of source GOPs, decoded GOPs, and adapted GOPs waiting for assignment to respective decoders, signal-adaptors, and encoders, respectively. If any of the counts exceeds a respective prescribed threshold, the orchestrator assembly instructs the multimedia source140, through the network interface2320, to pause transmission of the multimedia stream. When none of the counts exceeds a corresponding prescribed threshold while transmission of the multimedia stream is paused, the orchestrator assembly instructs the multimedia source, through the network interface, to resume transmission of the multimedia stream.

The content-processing assembly2750comprises a first content-access unit2710coupled to the network interface2320and the pool of decoders1420, a second content-access unit2720coupled to the pool of decoders1420and the pool of signal-adaptors1440, a third content-access unit2730coupled to the pool of signal-adaptors1440and the pool of encoders1460, and a fourth content-access unit2740, comprising a collating module, coupled to the pool of encoders1460and the network interface2320.

The orchestrator assembly comprises an orchestrator core2450coupled to three controllers. A first controller2451is coupled to the first content-access unit2710and the pool of decoders1420. A second controller2452is coupled to the second-content-access unit2720and the pool of signal adaptors1440. A third controller2453is coupled to the third content-access unit2730and the pool of encoders1460.

The network interface is further configured to affix a GOP identifier2020,2120,2220, to each source GOP of the multimedia stream and communicate the source GOP identifiers and corresponding metadata to the orchestrator assembly2350and to the content-processing assembly2750.

The first controller2451is configured to queue an identifier of each source GOP and, upon locating an available decoder, instruct the content-processing assembly to process content of a queued source GOP. The second controller2452is configured to queue an identifier of each decoded GOP and, upon locating an available signal adaptor, instruct the content-processing assembly to process content of a queued decoded GOP. The third controller2453is configured to queue an identifier of each adapted GOP and, upon locating an available decoder, instruct the content-processing assembly to process content of a queued adapted GOP.

According to an implementation, the pool of decoders1420comprises a first number, μ1, μ1≥1, of clusters1560of respective distinct decoders. The pool of signal-adaptors1440comprises a second number, μ2, μ2≥1, of clusters1570of respective distinct signal-adaptors. The pool of encoders1460comprises a third number, μ3, μ3−1, of clusters1580of respective distinct encoders.

Based on information within the transcoding request, an available decoder is selected from a compatible cluster1560of the first number of clusters, an available signal-adaptor is selected from a compatible cluster1570of the second number of clusters, and an available encoder is selected from a compatible cluster1580of the third number of clusters.

The disclosed method of multimedia transcoding comprises receiving, at a network interface2320, from a multimedia source240, a transcoding request and a multimedia stream. A hardware orchestrator2350assembly and a content-processing assembly2750perform requisite transcoding processes of the multimedia stream.

The multimedia stream comprises a video stream1900organized into source groups-of-pictures (GOPs) and corresponding metadata. The transcoding request indicates a first standard according to which the multimedia stream is formed and a second standard according to which an encoded stream is to be formed.

Multiple decoding workers1420of the pool of decoding workers are activated concurrently to decode different source GOPs, according to the first standard, to produce respective decoded GOPs;

Multiple signal-adaptation workers1440, of a pool of signal-adaptation workers, are activated concurrently to process different decoded GOPs of the respective decoded GOPs to produce respective adapted GOPs.

Multiple encoding workers1460, of a pool of encoding workers, are activated concurrently to encode different adapted GOPs of the respective adapted GOPs to produce respective encoded GOPs according to the second standard.

The network interface affixes a GOP identifier2020,2120,2220to each source GOP of the multimedia stream; and communicates the GOP identifiers to the orchestrator assembly and to the content-processing assembly.

The method segments the pool of decoding workers1420into a first number, μ1, μ1≥1, of distinct clusters1560of decoding workers, the pool of signal-adaptation workers1440into a second number, μ2, μ2≥1, of distinct clusters1570of signal-adaptation workers; and the pool of encoding workers1460into a third number, μ3, μ3≥1, of distinct clusters1580of encoding workers. Based on information within the transcoding request, the orchestrator assembly2350selects the multiple decoding workers from a respective cluster1560of decoding workers, selects the multiple signal-adaptation workers from a respective cluster1570of signal-adaptation workers, and selects the multiple encoding workers from a respective cluster1580of encoding workers.

The orchestrator assembly continually tracks a first count of GOPs waiting for processing at the respective cluster of decoding workers, a second count of decoded GOPs waiting for processing at the respective cluster of signal-adaptation workers, and a third count of adapted GOPs waiting for processing at the cluster of encoding workers.

If any of the first count, second count, or third count exceeds a respective prescribed threshold, the orchestrator assembly instructs the multimedia source140, through the network interface2320, to pause transmission of the multimedia stream.

If none of the first count, second count, and third count exceeds a corresponding prescribed threshold while transmission of the multimedia stream is paused, the orchestrator assembly instructs the multimedia source, through the network interface, to resume transmission of the multimedia stream.

The orchestrator assembly arranges successive encoded GOPs, of the respective encoded GOPs, according to an order of corresponding source GOPs of the multimedia stream.

In an alternate view, with a focus on selected clusters of resources applicable to a specific stream, the orchestrator assembly2350is coupled to a cluster1560of N1decoders, N1≥1, a cluster1570of N2signal adaptors, N2≥1, and a cluster1580of N3encoders, N3≥1 (FIG.6,FIG.7,FIG.18).

A 1:N1selector2710is configured to distribute contents of successive source GOPs to orchestrator-selected decoders to produce respective decoded GOPs. An N1:N2switching element2720is configured to distribute the respective decoded GOPs to orchestrator-selected signal adaptors to produce respective adapted GOPs. An N2:N3switching element2730is configured to distribute the respective adapted GOPs to orchestrator-selected encoders to produce respective encoded GOPs. An N3:1 selector2740coupled to a collating module is configured to collate the respective encoded GOPs to form a contiguous transcoded video stream.

Each decoder is compatible with a first standard, indicated in the transcoding request, according to which the multimedia stream is formed. Each encoder is configured to encode an output of any signal adaptor according to a second standard specified in the transcoding request.

The orchestrator assembly2350is configured to track a first count of source GOPs waiting for decoding at the cluster of N1decoders, a second count of decoded GOPs waiting for signal adaptation at the cluster of N2signal adaptors, and a third count of adapted GOPs waiting for decoding at the cluster of N3encoders.

If any of the first count, the second count, or the third count exceeds a respective prescribed threshold, the orchestrator assembly2350instructs the multimedia source140to halt transmission of the multimedia stream. If none of the first count, the second count, and the third count exceeds the prescribed threshold while transmission of the multimedia stream is halted, the orchestrator assembly instructs the multimedia source140to resume transmission of the multimedia stream.

The cluster of N1decoders is selected from a plurality of decoders based on the first standard and detailed characterization of content of the multimedia stream indicated in the metadata. The cluster of N2signal adaptors is selected from a plurality of signal adaptors based on requisite signal-adaptation processes indicated in the metadata. The cluster of N3encoders is selected from a plurality of encoders based on the second standard. Each of the N1decoders, N2signal adaptors, and N3encoders comprises a respective hardware processor coupled to a memory device storing software instructions, and a buffer holding intermediate data.

Thus, the disclosed technique of high-speed transcoding enables transcoding at a much higher speed in comparison with conventional techniques and eliminates the need to download and locally store the multimedia file to be transcoded, the produced transcoded file, or intermediary format. Any transient media format is streamed from one stage of transcoding to another in a continuous way, reducing the amount of RAM memory used as opposed to storing complete intermediate media before passing them on to the next stage.

It is noted that the methods described above adapt to GOPs of unequal durations, sizes, or numbers of frames per GOP within the same media.

Heterogenous Multifunctional Resources

FIG.40illustrates an example4000of heterogenous resources of a large-scale transcoding station where workers of any stage comprise a mixture of uni-functional and multi-functional workers with the multifunctional workers comprising uni-tasking worker and multitasking workers. A uni-functional worker performs a single task. A multi-functional worker is configured to perform more than one task. A multi-functional worker may be uni-tasking, capable to perform only one task at a time, or multi-tasking, capable to perform more than one task concurrently. In the illustrated example, the total number of tasks that a specific stage (decoding, signal processing, or encoding) may perform is eight, with the tasks individually identified as 0 to 7.

An identifier of a uni-functional worker has a leftmost digit of “0” and a following binary number4010indicating an index of a task. An identifier of a multi-functional worker has a leftmost digit of “1”, a following binary number4020indicating the maximum number of concurrent tasks, then a string4030of eight binary digits identifying tasks that a respective worker where a digit “1” in position p, 0≤p<8, indicates that a respective worker is configured to perform task p. In the case of a multifunctional uni-tasking worker, the binary number4020is “001” indicating that any of the tasks identified in the position-identified tasks of string4030may be performed one at a time.

In the case of a multifunctional multitasking worker, the binary number4020, which exceeds “001”, is the maximum number of tasks, selected from the tasks identified in the position-identified tasks of string4030, that may be performed concurrently. A binary number4020of “000” indicates that all of the position-identified tasks in string4030may be performed concurrently. For example:an identifier4020A “1.001.01001100” refers to a multifunctional uni-tasking worker configured to perform any of the position-identified tasks in string “01001100”, which are the three tasks of indices 1, 4, and 5;an identifier4020B “1.000.01000100” refers to a multifunctional worker configured to concurrently perform all of the position-identified tasks in string “01000100”, which are the two tasks of indices 1 and 5;an identifier4020C “1.010.00011100” refers to a multifunctional worker configured to concurrently perform any two tasks of the position-identified tasks in string “00011100”, which are the three tasks of indices 3, 4, and 5; andan identifier4020D “1.100.01110011” refers to a multifunctional worker configured to concurrently perform any four tasks of the position-identified tasks in string “01110011”, which are the five tasks of indices 1, 2, 3, 6, and 7.

FIG.41illustrates a preferred order4100of selection of the heterogenous workers ofFIG.40, starting with a set4110of uni-functional worker, then a set4120of multifunctional uni-tasking workers sorted according to an ascending order of versatility indicators, and a set4130of multifunctional multitasking workers sorted according to an ascending order of versatility indicators. The versatility indicator of a worker type is the number of tasks that a worker is configure to perform whether one at a time or concurrently. In the example ofFIG.40, the versatility of a worker type is the number of “1” digits within a corresponding string4030. Thus, the versatility indicators of the nine multifunctional uni-tasking worker types (middle column from top, are {1, 2, 2, 3, 4, 5, 6, 6, 8}, respectively, and the versatility indicators of the twelve multifunctional multitasking worker types (right column, from Top) are {2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 8}, respectively.

FIG.42illustrates a first example4200of a mixture of uni-functional and multifunctional worker types4210. Seven worker types, indexed as 0 to 6. are indicated with corresponding:numbers4220of workers per worker types;indices4230of tasks that a respective worker is configured to perform; andidentifiers4240of individual workers.

In the example ofFIG.42, the total number of workers is 16, identified as 0 to 15. There are two workers (0, 1) of worker-type 00, two workers (2, 3) of worker-type 01, two workers (4, 5) of worker-type 02, three workers (6, 7, 8) of worker-type 03, three workers (9, 10, 11) of worker-type 04, two workers (12, 13) of worker-type 05, and two workers (14, 15) of worker-type 06. A worker of worker-type 00 performs only a task of index 0, a worker of worker-type 03 may perform task 0 and/or task 1. A worker of worker-type 06 may perform all, or any subset of tasks 0, 1, and 2.

FIG.43illustrates a second example4300of a mixture of uni-functional and multifunctional worker types. Fifteen worker types indexed as 00 to 14 are indicated with corresponding numbers4220of workers per worker types, indices4230of tasks that a respective worker is configured to perform, and identifiers4240of individual workers. In the example ofFIG.43, the total number of workers is 40, identified as 0 to 39. A number of workers4230designated as a “*” signifies that workers of a corresponding type are not yet provided. There are two workers (11, 12) of type 06, each configured to perform tasks 0 and 3, five workers (15, 16, 17, 18, 19) of type 09, each configured to perform tasks 2 and 3, and five workers (35, 36, 37, 38, 39) of type 14, each configured to perform tasks 0, 1, 2, and 3.

Workers of types 00 to 03, are uni-functional workers of a versatility indicator of 1. Workers of types 04 to 09 are multifunctional workers of a versatility indicator of 2. Workers of types 10 to 13 are multifunctional workers of a versatility indicator of 3. Workers of type 14 are multifunctional workers of a versatility order of 4.

FIG.44is a tabulation4400of worker types relevant to each task of each stage, the worker types being sorted in an ascending order according to worker's versatility. The numbers of worker types of stage-1 (the decoding stage), stage-2 (signal-processing stage-2), and stage-3 (the encoding stage) are 14, 9, and 12, respectively. A worker-type is stage-specific; a worker of type 02 of stage-1 performs a decoding-related task while a worker of type 02 of stage-3 performs an encoding task.

FIG.45illustrates a data structure4500maintaining stage-specific workers data for facilitating selection of workers of a stage for a case where all workers are uni-tasking workers in a transcoding station employing a mixture of uni-functional and multifunctional uni-tasking workers. The symbol “*” (reference4550) indicates a worker type that has not yet been provided. For each worker type, a number of provisioned workers, a number of available (free) workers, a versatility indicator, and respective task indices are indicated. In the example ofFIG.45, a worker of any of types {00, 04, 05, 06, 10, 11, 12, 14} is configured to perform task-0, a worker of any of types {01, 04, 07, 10, 11, 13, 14} is configured to perform task-1, and so on. The worker types are listed in an ascending order of versatility. If each worker is configured to perform one task at a time, then a worker of any worker type for which the number of available workers is greater than zero may be allocated to a task under consideration. Preferably, a worker is selected from a worker group of a worker type of least versatility in order to increase the availability of workers of higher versatility for forthcoming allocations.

FIG.46illustrates a data structure4600for facilitating worker selection in a transcoding station employing a mixture of uni-functional and multifunctional workers of both uni-taking and multitasking capabilities. The data4620under the field of “available resources” is the product of the number of workers of a respective worker type and a corresponding maximum number of concurrent tasks. For example, a worker of a worker group of worker-type 11 can concurrently perform any two of three tasks (task-0, task-1, and task-3). Thus, the maximum number of available resources pertinent to worker-type 11 is 16.

The data4640under the fields of “number of task-specific available workers” for a specific task (of task type-0, task-type-1, task-type-2, or task-type-3) is the number workers of a respective worker type that are not performing the specific task and, therefore, may be available. Initially, the number of task-specific available workers of a work group of any work type is the number of workers of the work group. For example, the number4640A of available workers of work-type 4 configured to perform task 0 and/or task 1 is the number or workers of type 4.

The “available resources” and “number of task-specific available workers”, are time varying. The values indicated inFIG.46correspond to the initial state before any worker allocation takes place while values indicated inFIG.47correspond to the state of the transcoding station at a later time instant.

FIG.47illustrates exemplary entries4700of data structure4600in operation, corresponding to a particular stage, indicating a snapshot of availability of workers of the mixture of workers ofFIG.46. To allocate a worker of a candidate work-group (of a work type) to perform a task of a specific task-type to a new GOP, two conditions must be met: the number of task-specific workers and the number of available resources of the candidate work group must be greater than zero. For example, to allocate a worker to perform a task of task-type-2, candidate worker groups of worker types 07, 09, 12, 13, and 14 are examined. For worker-group 07, the number of task-specific available workers is zero. For worker-group-13, the number of available resources is zero. Hence, a worker may be selected from any of worker groups 09, 12, or 14 with corresponding versatility indicators of 2, 3, and 4, respectively. As illustrated inFIG.41, the first available worker in the versatility-sorted list of worker types is preferred.

FIG.48illustrates processes4800of worker selection of heterogenous workers implemented as a stage controller2451,2452, or2453. Process4810selects the first worker type applicable to a current task type as a current worker type. Process4820branches to process4825if the number of available resources4620for the current worker type exceeds zero, or branches to process4865otherwise. Process4825branches to process4830if the number of available workers for the current worker type exceeds zero, or branches to process4865otherwise. Process4830identifies a candidate worker, within the group of workers of the current worker type, satisfying two conditions: the candidate worker is not executing a task of the current task type; and the number of tasks that the candidate worker is executing is less than a respective multi-tasking limit.

Process4840branches to process4865if the candidate worker does not meet the above two conditions, or branches to process4850which allocates the candidate worker to a new GOP. Process4860then updates the state of the allocated candidate worker and reports the identifier of the allocated candidate worker.

Process4865determines if all worker-types (hence all worker groups) that are applicable to the current task type have been considered. If so, process4880reports unavailability of an appropriate worker; otherwise process4870is activated to select a subsequent worker-type that is applicable to the current task type as the current worker type then revisits process4820.

FIG.49illustrates worker-type-specific tasks4900that may be performed at each of uni-functional worker-type and multi-functional worker types used inFIG.43,FIG.46, andFIG.47.

FIG.50illustrates exemplary worker-selection scenarios5000for the worker-group of worker-type 14 (FIG.46,FIG.47) which comprises five workers 35, 36, 37, 38, and 39, as indicated inFIG.33, each configured to perform any of 4 tasks (versatility=4) but with a multi-tasking limit of 2 (at most two concurrent tasks per worker). Scenarios5010,5020,5030, and5040correspond to cases where the work load of the worker group comprises three tasks of task-type 0, three tasks of task-type-1, two tasks of task-type-2, and two tasks of task type-3. Any three workers of the worker group may perform the three tasks of task-type 0, any two workers of the worker-group may perform the two tasks of task-type 2, etc.

FIG.51illustrates a procedure5100for constructing a data structure (such as the exemplary data structure ofFIG.36) for facilitating allocation of workers. Process5110identifies a number μj, μj>1, of task types, labelled as 0 to (μj−1) performed in stage j, 0≤j<Q, Q being a number of stages (Q=3 in the structure ofFIG.6,FIG.7, andFIG.8). In the structure ofFIG.33,FIG.36andFIG.47, μj=4.

Process5120identifies a number of individual workers of stage-j, indexed sequentially as 0 to (∧j−1), in steps of 1, which are provisioned for stage-j. In the structure ofFIG.33, ∧j=40. Process5130sorts the ∧j workers into a number Ωj, Ωj>1, of types of workers, indexed sequentially as 0 to (Ωj−1) in steps of 1. In the structure ofFIG.33,FIG.36, andFIG.47, Ωj=15. Process5140tracks the number of free workers of each group of workers of a same worker-type.

Process5150determines a versatility indicator of each type of workers according to a count of tasks that a worker is equipped to implement. Process5160sorts the Ωjtypes of workers in an ascending order according to versatility indicators to produce a sorted list of types of workers. In the structure ofFIG.33andFIG.36, worker-types 00 to 03 have a versatility of 1, worker-types 04 to 09 have a versatility of 2, worker-types 10 to 13 have a versatility of 3, and worker-type 14 has a versatility of 4. For each task-type k, 0≤k<μj, of stage-j, process5170lists the types of workers equipped to implement a task of type k and indicates corresponding task-type-specific number of free workers.

FIG.52illustrates a procedure5200for allocating tasks to workers for a case of a transcoding station employing a mixture of uni-functional and multifunctional workers where all workers are uni-tasking workers (such as in the case ofFIG.45). Process5210receives a GOP and corresponding requisite tasks from a previous stage and initializes a “set of selected workers” as an empty set. Process5220selects the first requisite task as a current task. Process5230finds the first worker type that has at least one free worker. If no free worker is found, process5240branches to process5250which reports the set of selected workers, if any, with an indication of unassigned tasks. Otherwise, process5260adds an identifier of the free worker to the set of selected workers and reduces the number of free workers of the respective work group. If all requisite tasks have been assigned, process5270branches to process5290which reports the set of selected workers for the requisite tasks. Otherwise, process5270branches to process5280which selects a subsequent requisite task and revisits process5230.

FIG.53illustrates a procedure5300for allocating workers to tasks for a case of a transcoding station employing a mixture of uni-functional and multifunctional workers with workers of both uni-tasking and multicasting capabilities. As in the procedure ofFIG.52, process5210receives a GOP and corresponding requisite tasks from a previous stage and initializes a “set of selected workers” as an empty set. Process5220selects the first requisite task as a current task.

Process5330finds the first worker type that has at least one free worker based on respective task-specific available workers and available resources3620. If no free worker is found, process5340branches to process5350which reports the set of selected workers, if any, with an indication of unassigned tasks. Otherwise, process5340branches to process5360which adds an identifier of the free worker to the set of selected workers and reduces the number of task-specific available workers and the number of available resources of the respective work group. If all requisite tasks have been assigned, process5370branches to process5390which reports the set of selected workers for the requisite tasks. Otherwise, process5370branches to process5380which selects a subsequent requisite task and revisits process5330.

FIG.54is an overview5400of a transcoding station configured to concurrently transcode multiple streams using a shared orchestrator core. Dual links5420connect the network interface2320to network120. The network interface directs overall control data5421from all connecting multimedia sources140to an orchestrator assembly5430having a single orchestrator core configured to handle multiple streams, and overall content data5422from all connecting multimedia sources140to a pool5460of overall stage-1 workers, handling all streams, which transfer processed content of all streams to a pool5470of stage-2 workers handling all streams which, in turn, transfer processed content of all streams to a pool5480of stage-3 workers handling all streams.

FIG.55is an overview5500of a transcoding station configured to concurrently transcode multiple streams using a set5540of K orchestrator assemblies5530(1) to5530(K), K>1. The set of K orchestrator assemblies control content processing and content transfer through dual control paths5550between the set5540of orchestrator assemblies and the pools of shared workers5460,5470, and5480. The number of multimedia streams that may be transcoded concurrently is determined dynamically according to execution-duration measurements.

FIG.56further details the transcoding station ofFIG.55. Each of orchestrator assemblies5530(1) to5530(K) comprises a respective orchestrator core5650coupled to respective stage controllers5651,5652, and5653, of stages 1 to 3, respectively. Orchestrator-assembly5530(j) comprises orchestrator core5650(j) and stage controllers5651(j),5652(j), and5653(j), 1≤j≤K.

FIG.57illustrates a shared content-processing assembly5700for the transcoding station ofFIG.54including a first switch2710for distributing incoming GOPs from multimedia sources140to first-stage workers5760, a second switch2720for distributing decoded GOPs to second-stage workers5770, a third switch2730for distributing processed GOPs to third-stage workers5780, and a fourth switch2740for distributing encoded (transcoded) GOPs to respective clients180through the network interface320.

The workers of each stage are adaptively allocated to any stream of a set of concurrently transcoded streams. Stage-1 workers5760include a first number of decoders which may be dynamically partitioned among multiple independent GOP streams. Likewise, stage-2 workers5770include a second number of VPUs which may be partitioned, and stage-3 workers5780include a third number encoders which may be partitioned.

FIG.58illustrates policies5800for allocation of workers of a specific worker type to tasks of multiple streams of the transcoding station ofFIG.54orFIG.55. Individual workers5850of a pool5860of workers of a specific worker type may be allocated to handle GOPs of different streams according to any of sharing policies.

According to a first worker-allocation policy5810, the pool5860of workers may be segmented into stream-specific groups with an additional group of floating workers that may be adaptively allocated to any of the streams based on workload fluctuation. In the illustrated example, the pool comprises 20 workers5850serving three streams labeled stream-1, stream-2, and stream-3. A set5811of five workers5850is reserved for stream-1, a set5812of three workers is reserved for stream-2, and a set5813of six workers is reserved for stream-3. Workers of the remaining group5814of six workers may be individually allocated to any of the three streams according to time-varying processing requirement; any released worker of group5814becomes available to any stream.

According to a second worker-allocation policy5820, any worker of the pool5860of workers may be allocated to any of the streams and when released may be allocated to any other stream. As illustrated, at some time instant, the workers5850may be allocated as indicated in pattern5840with five workers, individually referenced as5821allocated to tream-1, three workers, individually referenced as5822allocated to tream-1, six workers, individually referenced as5823allocated to tream-3. Unassigned or released workers, individually referenced as5824may be individually allocated to any stream.

FIG.59is an overview5900of a large-scale transcoding station concurrently receiving at large-scale network interface5920multiple incoming multimedia streams and producing multiple transcoded streams per incoming stream according to different standards. With K parallel streams, K>1, received from different multimedia sources140, each stream is processed using a respective pool of stage-1 workers, pool of stage-2 workers, and pool of stage-3 workers. Additionally, parallel stage-3 pools may be needed for at least one stream to handle the case where a transcoded stream is directed to receivers obeying different standards. As illustrated, stage-1 pools of workers5941to5961are indexed as (1, j), stage-2 pools of workers5942to5962are indexed as (2, j), and stage-3 pools of workers5943to5963are indexed as (3, j, Πj), for a stream of index j, 1≤j≤K, where Πj is a number parallel stage-3 pools of workers for stream-j.

System Analytics

FIG.60illustrates a method6000of measuring task-execution durations using both training data and operational data. The method may be implemented at orchestrator core2450or at any, or all, of orchestrator cores5530(1) to5530(K). Process6010identifies characteristics (such as GOP sizes, frame rates, standard according to which incoming GOPs are formed, standard according to which outgoing GOPs are to be formed, etc.) of GOPs handled at each of the three stages (decoding stage, video-signal-processing stage, and encoding stage). Process6020measures task-execution time duration at each stage for each worker type. Process6030applies an appropriate optimal-estimation method to determine parameters of an appropriate parametric model relating task-execution duration to GOP characteristics and worker types. Process6040estimates a requisite number of workers per stage based on the measurements.

FIG.61illustrates tracked data6100relevant to stage-1 including time measurements corresponding to specific stage-1 metadata. The metadata includes frames/second of received GOPs, number of frames per GOP, number of pixels per raw frame, sizes of compressed GOPs.

FIG.62illustrates tracked data6200relevant to stage-2 including time measurements corresponding to specific stage-2 metadata. The metadata includes frames/second of processed GOPs, and requisite signal-processing tasks.

FIG.63illustrates tracked data6300relevant to stage-3 including time measurements corresponding to specific stage-3 metadata. The metadata includes frames/second of transcoded GOPs, number of frames per transcoded GOP, and sizes of compressed transcoded GOPs.

Methods of the embodiment of the invention are performed using one or more hardware processors, executing processor-executable instructions causing the hardware processors to implement the processes described above. Computer executable instructions may be stored in processor-readable storage media such as hard disks, Flash ROMS, non-volatile ROM, and RAM. A variety of processors, such as microprocessors, digital signal processors, and gate arrays, may be employed.

Systems of the embodiments of the invention may be implemented as any of a variety of suitable circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof. When modules of the systems of the embodiments of the invention are implemented partially or entirely in software, the modules contain a memory device for storing software instructions in a suitable, non-transitory computer-readable storage medium, and software instructions are executed in hardware using one or more processors to perform the techniques of this disclosure.

It should be noted that methods and systems of the embodiments of the invention and data streams described above are not, in any sense, abstract or intangible. Instead, the data is necessarily presented in a digital form and stored in a physical data-storage computer-readable medium, such as an electronic memory, mass-storage device, or other physical, tangible, data-storage device and medium. It should also be noted that the currently described data-processing and data-storage methods cannot be carried out manually by a human analyst, because of the complexity and vast numbers of intermediate results generated for processing and analysis of even quite modest amounts of data. Instead, the methods described herein are necessarily carried out by electronic computing systems having processors on electronically or magnetically stored data, with the results of the data processing and data analysis digitally stored in one or more tangible, physical, data-storage devices and media.