Redundancy control in streaming content encoder pools

Systems and methods are described to enable synchronized encoding of streaming audio or video content between multiple encoders, in a manner that provides for redundancy of the system to vary based on a demand for the output content. End user devices or content distribution systems can monitor how content is output on end user devices, and report such output to a content encoding system. The encoding system can then redundancy provided for streaming content based on the demand by end users. Streams that are in high demand can be processed with high redundancy among devices that provide seamlessly interchangeable content, thus reducing the likelihood of perceived failure for such streams. Streams that are in low demand can be processed with low redundancy, reducing the computing resources used to process the stream while minimizing the overall impact of a processing failure, should one occur.

BACKGROUND

Service providers or content creators (such as businesses, artists, media distribution services, etc.) can employ interconnected computing devices (e.g., within data centers) to deliver content to users or clients. In some instances, these computing devices may support traditional content distribution systems, such as by creating, modifying, or distributing streaming television or radio content. In other instances, these computing devices may serve to replicate or replace prior content distribution systems. For example, data centers can provide network-based streaming video or audio content in a manner similar to traditional television or radio networks. This content is sometimes referred to as “internet television” or “internet radio,” respectively. The content provided by these distribution systems (e.g., both traditional and computing network-based) may be pre-recorded, or live. Often, where computing devices are used to facilitate either traditional or network-based distribution systems, specialized software is used to replace or replicate functionality of dedicated hardware devices. For example, software applications may be used to encode and package a data stream containing live video content, thus reducing or eliminating the need for dedicated hardware to perform these functions. Because of the flexibility of software-based solutions, a single computing device may be utilized to generate content for both traditional and network-based generation systems.

DETAILED DESCRIPTION

Generally described, the present disclosure relates to providing streaming content over a communication network, by utilizing pools of redundant or cooperative content encoders configured according to a dynamic redundancy model, which can vary according to demand or consumption of the streaming content. More specifically, the present disclosure relates to a streaming content system that includes multiple content encoders, sometimes referred to herein as a “pool.” A pool management system that can vary the number and configuration of encoders within a pool based on consumption or demand for streaming content, such that the redundancy or expected resiliency can be increased as demand for the streaming content increases. For example, aspects of the present disclosure can encode content streams characterized as popular, or in high demand, with high redundancy, such that errors or failures within individual encoders in the encoder pool result in little or no downtime from the perspective of the clients receiving requested content. Conversely, aspects of the present disclosure can encode content streams characterized as unpopular, or in lower demand, with little or no redundancy, minimizing computing resources used to encode such content. Moreover, embodiments of the present disclosure can monitor a level of consumption of a content stream to determine an appropriate level of redundancy for the stream, and modify configurations associated with an encoder pool for the content stream to achieve the appropriate level of redundancy, such that little or no human intervention is required to configure or modify encoder pool redundancy.

Illustratively, embodiments of the present disclosure can utilize virtual machines (individual instances of which can be referred to as virtual machine instances) to encode and package content streams, and to distribute such content streams to end users. A pool management system can operate to monitor consumption (or delivery) of the content streams, and to select a level of redundancy for the content stream based on that consumption. Thereafter, the pool management system can interact with the virtual machines within an encoding pool (or computing devices acting as host to the virtual machines) to modify the number or configuration of the encoder pool to implement the selected level. Level of redundancy may be measured, for example, by the number of computing devices (e.g., virtual machines) encoding content.

Illustratively, individual virtual machines may be associated with an expected failure rate (e.g., expressed as mean-time to failure or other statistical values). Additionally, an expected failure rate for a content stream being encoded by a pool of N virtual machines may be measured with respect to the expected failure rate of the virtual machines in the pool, such as by determining the probability that each virtual machine would fail concurrently. By increasing the number of virtual machines, the expected failure rate for the content stream can therefore be decreased. Additionally or alternatively, an expected failure rate may be measured by an expected downtime of a content stream due to failure. For example, where only a single device is encoding a content stream, the expected failure rate for the stream may be measured by the time required to start encoding after the device fails, such as by rebooting the failed device or by provisioning a new device to resume encoding. In some instances, the time required resume encoding a stream is dependent on the state of virtual machines in a pool. For example, one or more virtual machines in a pool may be maintained in a “warmed” state, such that the virtual machine is running (e.g., loaded into an operating system) and configured with necessary software to encode content, but is not actually processing content. By having one or more virtual machines in a “warmed” state, downtime required to recover from a failed device can be minimized, as little or no time is required to boot the machine, load an operating system, etc.—instead, the device can simply start to encode content from a point at which a failed device stopped.

In other embodiments, expected failure rates can be measured both with respect to a number of devices encoding content and with respect to a time required to resume encoding after a device failure. For example, expected failure rates may reflect the expected time required to return an encoder pool to producing n copies of a content stream after a device failure.

While high levels of redundancy are generally desirable for content encoding, excessive redundancy is associated with increased computing resources, and thus reduced efficiency of computing systems overall. Put in other terms, while having an input content stream encoded into hundreds of identical outputs may largely eliminate downtime in the case of failure, operating such a system would require large amounts of processing power, memory, network resources, electricity, etc. Such expenditures are typically not warranted for all content streams, particularly those that are in lower demand. While one solution might be to manually configure the redundancy on a stream-by-stream basis, according to demand for the stream, the variability and volatility of demand makes such configuration difficult or impossible. Specifically, because content streams are often widely distributed (e.g., via a global area network) in a real-time manner, it can be difficult or impossible to accurately forecast demand for a given stream. Moreover, the delay caused by manually monitoring a streams popularity and adjusting the level of redundancy of a stream may significantly reduce the efficacy of such adjustments, as demand may vary much more quickly than redundancy could be manually updated. The present disclosure provides an approach to providing a desired level of redundancy according to the demand or popularity of the content stream, thereby increasing the resiliency of popular content streams while reducing the computing resources required to encode less popular content streams.

In some instances, a particular input content, such as a live news feed, may be encoded into multiple output content stream versions (e.g., a low-bandwidth version, a standard definition version, a high definition version, etc.). To manage levels of redundancy in encoder pools, some embodiments of the present disclosure may monitor demand across different versions, and vary the number of devices encoding each version. Illustratively, where a standard definition version has a relatively low demand, the pool management system disclosed herein may reallocate computing devices from encoding the standard definition version to encoding a different version. In some instances, the total number of devices used to encode all versions may remain static during reallocation of devices between versions. In this manner, resiliency can be increased for those versions of a content stream in highest demand, increasing the overall efficiency of the system at the same or similar level of computing resource use.

While aspects of the present disclosure are discussed with relation to encoder pools, content encoding systems may in some instances include additional or alternative pools of computing devices that are used to provide encoded content. For example, content encoding systems may include a pool of content packagers, which operate to package encoded content into a format readable by end user computing devices. Aspects of the present disclosure may additionally or alternatively be used to modify the number and configuration of content packager pools.

In some instances, a content encoding system may configure encoders within an encoder pool to produce identical or seamlessly interchangeable content, such that even if a device producing a content stream fails, a downstream component such as a content packager or end user may obtain an alternative content stream, with little or no perceptible variation in the content stream. To produce identical or seamlessly interchangeable content, encoders within a content stream may use a communications protocol, enabling the encoders to communicate with one another (or a centralized device) to achieve a shared encoding state. Systems and methods for implementing such a communications protocol among a pool of encoders are described in U.S. patent application Ser. No. 15/194,347, filed Jun. 27, 2016, and entitled SYNCHRONIZATION OF MULTIPLE ENCODERS FOR STREAMING CONTENT,” (the '347 application), the entirety of which is incorporate by reference herein.

As will be appreciated by one of skill in the art in light of the present disclosure, the embodiments disclosed herein improves the ability of computing systems, such as content streaming systems, to deliver content to users in a resilient and/or cooperative manner. Specifically, aspects of the present disclosure improve the ability of content streaming systems to provide redundant or cooperative encoding to an extent appropriate for a content stream, while minimizing inefficiencies created by over-use of redundant encoding systems. Moreover, the presently disclosed embodiments address technical problems inherent within computing systems; specifically, the unreliability of non-redundant systems, the limited ability of a single device to encode content according to a variety of formats or parameters, the inefficiencies of redundancy systems for low demand content, and the impossibility or impracticality of manually altering redundancy based on demand levels. These technical problems are addressed by the various technical solutions described herein, including the use of a pool manager to monitor consumption of a content stream by end users, and to vary the number or configuration of encoding devices according to the demand, to reach an appropriate level of redundancy. Thus, the present disclosure represents an improvement on existing content streaming systems and computing systems in general.

The foregoing aspects and many of the attendant advantages of the present disclosure will become more readily appreciated as the same become better understood by reference to the following, when taken in conjunction with the accompanying drawings.

FIG. 1is a block diagram depicting an illustrative logical network environment100including multiple content output devices102, content providers104, and content distribution systems120in communication with a streaming content delivery system110via a network106. While the content output devices102, the content providers104, and the content distribution systems120are shown as grouped withinFIG. 1, the content output devices102, content providers104, and content distribution systems120may be geographically distant, and independently owned or operated. For example, the content output devices102could represent a multitude of devices in various global, continental, or regional locations accessing the content streaming system110. Further, the content providers104could represent a multitude of related or distinct parties that have associated with the content streaming system110to provide streaming content to the content output devices102. Still further, the content distribution systems120could represent a multitude of related or distinct parties enabling distribution of content from the content streaming system110to the content output devices102. While shown as distinct, any one or more of the content streaming system110, content output devices102, content providers104, or content distribution systems120may be operated by a common entity, or by a common computing device. Similarly, each of the components of the content streaming system110may be located within geographically diverse areas. For example, while shown as a collection, the content encoders114may in some instances include devices in a number of geographic regions (e.g., corresponding to cities, states, countries, continents, etc.), and thus Accordingly, the groupings of content output devices102, content providers104, content streaming system110, and content distribution systems120withinFIG. 1is intended to represent a logical, rather than physical, grouping.

Network106may be any wired network, wireless network, or combination thereof. In some instances, network106may be a telecommunications network, such as a personal area network, local area network, wide area network, cable network, satellite network, cellular telephone network, or combination thereof operating via the internet protocol (IP). In other instances, the network106may be or may include other network types, such as television networks or radio networks. In the example environment ofFIG. 1, network106is a global area network (GAN), such as the Internet. Protocols and components for communicating via the other aforementioned types of networks are well known to those skilled in the art of electronic communications and thus, need not be described in more detail herein. While each of the content output devices102, content providers104, content distribution systems120, and content streaming system110is depicted as having a single connection to the network106, individual components of the content output devices102, content providers104, content distribution systems120, and content streaming system110may be connected to the network106at disparate points. Accordingly, communication times and capabilities may vary between the components ofFIG. 1. While shown inFIG. 1as a single network, the network106may represent different networks interconnecting the different components ofFIG. 1. For example, the network106may represent a first network (e.g., the Internet) that interconnects the content streaming system110and the content distribution systems120, and a second network (e.g., a television broadcasting network) that interconnects the content distribution system120and the content output devices102.

Content output devices102may include any number of different devices configured to output content from the content streaming system100(e.g., directly or via the content distribution systems120). For example, individual content output devices102may correspond to computing devices, such as a laptop or tablet computer, personal computer, wearable computer, server, personal digital assistant (PDA), hybrid PDA/mobile phone, mobile phone, electronic book reader, set-top box, camera, digital media player, and the like. As a further example, individual content output devices102may correspond to televisions, radios, or other output devices (e.g., which may or may not include or correspond to computing devices). Each content output device102may include hardware and/or software enabling the reception and output of streaming content, including dedicated playback hardware, dedicated software (e.g., specially programmed applications), and general purpose software (e.g., web browsers) capable of outputting streaming content (e.g., by downloading the content directly, downloading a web page including the content, etc.).

Content providers104may include any computing device owned or operated by an entity that provides content to the content streaming system110for subsequent transmission to content output devices102. For example, content providers104may include servers hosting streaming audio or video, or may include recording devices transmitting content to the content streaming system110(e.g., digital video recorders).

Content distribution systems120can include any computing device owned or operated by an entity that assists in delivery of content from the content streaming system110to the content output devices102. Illustratively, the content distribution systems120may include network-based content delivery networks (“CDNs”) that provide a geographically-diverse set of points of presence (“POPs”) from which content output devices102may obtain and output content. As a further illustration, the content distribution systems120may include distribution points in traditional networks, such as a television station, cable provider station, radio station, etc.

The content streaming system110can include a variety of components and devices configured to process streaming content obtained from a content provider and make processed content available to the content output devices102and/or the content distribution systems120. Specifically, the content streaming system110can include a content ingestor111configured to obtain a stream of content (e.g., live content) from a content provider104, and to provide that content to a pool112of content encoders114for encoding. The content provided to the content ingestor111may be in “raw,” uncompressed or analog format, or in a format that is otherwise required to be “encoded” prior to delivery to the content output devices102. Accordingly, the content ingestor111may pass the stream of content onto one or more content encoders114, which can encode the content into one or more formats accepted by the content distribution systems120or the content output devices102. While the term “encode” is used herein to discuss the processing of the content encoders114, use of this term is not intended to limit that processing to conversion of analog to digital formats. Rather, the content encoders114may in some instances process digital content to convert that digital content into another digital format (sometimes referred to as “transcoding”), and the term “encode” is intended to encompass to such conversions. As discussed above, in order to provide redundant or cooperative encoding of content (e.g., to provide resiliency or adaptive-quality streaming), multiple content encoders114may be configured to encode the stream of content received from the content ingestor111according to the same or to interchangeable parameters.

After encoding the content, each content encoder114may provide the encoded content to a pool118of content packagers116, which may package the content into a container format accepted by the content distribution systems120and/or content output devices102. The content packagers116can then distribute the content to the content distribution systems120, which may in turn distribute the content to content output devices102for decoding and output (e.g., display). Additionally or alternatively, the content streaming system110may distribute the content directly to the content output devices102.

In accordance with the present disclosure, the content streaming system110further includes a pool manager119, which can include a computing device configured with computer-executable instructions to manage the configuration of the encoder pool112or the packager pool118, such as by alternating the number of devices within the encoder pool or altering the state of such device (e.g., as in a fully operational state, encoding content, or as in a warmed state, loaded and ready but not yet encoding content). As will be described below, the pool manager119can operate to obtain information regarding demand for a content stream (or a version of a content stream) from content output devices102or content distribution systems120, and to vary the configuration of the encoder pool112or packager pool118based on that information. For example, the pool manager119can modify the encoder pool112to increase redundancy and resiliency for high demand content streams, while reducing redundancy or resiliency for low demand content streams (e.g., to reduce the computing resources used for such low demand streams). As a further example, the pool manager119may use demand information for a content stream to modify an arrangement or location of content encoders112within an encoder pool114or content packagers116within a packager pool118, such that encoders112or packagers116are instantiated in a location nearby (either in terms of physical or network distance) to content output devices102or devices of content distribution systems120receiving encoded content. As yet another example, the pool manager119may utilize demand information to vary the configuration of individual devices within an encoder pool112or packager pool118. For example, where such devices are implemented as virtual machines, the pool manager119may alter the amount of host computing resources (e.g., processing power, memory, network bandwidth, etc.) available to the virtual machine.

In some instances, a pool manager119may modify an encoder pool112or packager pool118(or devices within the respective pools112and118) according to additional or alternative criteria. For example, the pool manager119may modify a number or configuration of devices within an encoder pool112or packager pool118based on a forecasted demand during a given time (e.g., increasing the number of devices in an encoder pool112or packager pool118during peak viewing hours, decreasing the number of devices during other hours). As an additional example, the pool manager119may modify a location of devices within an encoder pool112or packager pool118to locate those devices nearby to output devices102or devices of content distribution systems120that are expected to experience higher demand (e.g., to locate devices within the encoder pool112or packager pool118within a time zone during peak viewing periods for that time zone).

It will be appreciated by those skilled in the art that the content streaming system110may have fewer or greater components than are illustrated inFIG. 1. In addition, the content streaming system110could include various web services or peer-to-peer network configurations. Thus, the depiction of the content streaming system110inFIG. 1should be taken as illustrative. In some embodiments, components of the content streaming system110, such as the content ingestor111, the content encoders114, the content packagers116, and the pool manager119may be executed by one more virtual machines implemented in a hosted computing environment. A hosted computing environment may include one or more rapidly provisioned and released computing resources, which computing resources may include computing, networking and/or storage devices. A hosted computing environment may also be referred to as a cloud computing environment. In other embodiments, components of the content streaming system110, such as the content ingestor111, the content encoders114, the content packagers116, and the pool manager119may be implemented within an environment of a content provider104(e.g., on the premises of a content provider104). In instances where the content streaming system provides multiple versions of a content stream, logical groupings of the encoder pool112or packager pool118may vary. For example, the content streaming system110may include multiple distinct pools for each content version output, or may include a single pool outputting each content version. While embodiments are described with respect to a single content stream (which may be output in multiple versions), the content streaming system110and the components shown therein may be configured to process multiple content streams. Thus, the configuration shown withinFIG. 1is intended to be illustrative, and not limiting

FIG. 2depicts one embodiment of an architecture of a server200that may implement the pool manager119described herein. The general architecture of server200depicted inFIG. 2includes an arrangement of computer hardware and software components that may be used to implement aspects of the present disclosure. As illustrated, the server200includes a processing unit204, a network interface206, a computer readable medium drive207, an input/output device interface220, a display222, and an input device224, all of which may communicate with one another by way of a communication bus. The network interface206may provide connectivity to one or more networks or computing systems, such as the network106ofFIG. 1. The processing unit204may thus receive information and instructions from other computing systems or services via a network. The processing unit204may also communicate to and from memory210and further provide output information for an optional display222via the input/output device interface220. The input/output device interface220may also accept input from the optional input device224, such as a keyboard, mouse, digital pen, etc. In some embodiments, the server200may include more (or fewer) components than those shown inFIG. 2. For example, some embodiments of the server200may omit the display222and input device224, while providing input/output capabilities through one or more alternative communication channel (e.g., via the network interface206).

The memory210may include computer program instructions that the processing unit204executes in order to implement one or more embodiments. The memory210generally includes RAM, ROM, or other persistent or non-transitory memory. The memory210may store an operating system214that provides computer program instructions for use by the processing unit204in the general administration and operation of the server200. The memory210may further include computer program instructions and other information for implementing aspects of the present disclosure. For example, in one embodiment, the memory210includes user interface software212that generates user interfaces (and/or instructions therefor) for display upon a computing device, e.g., via a navigation interface such as a web browser installed on the computing device. In addition, memory210may include or communicate with one or more auxiliary data stores, such as data store120, which may correspond to any persistent or substantially persistent data storage, such as a hard drive (HDD), a solid state drive (SDD), network attached storage (NAS), a tape drive, or any combination thereof.

In addition to the user interface module212, the memory210may include pool management software216that may be executed by the processing unit204. In one embodiment, the pool management software216implements various aspects of the present disclosure, e.g., managing the number and configuration of computing devices within encoder pools or packager pools. While the content encoder software216is shown inFIG. 2as part of the server200, in other embodiments, all or a portion of the software may be implemented by alternative computing devices within the content streaming system110, such as virtual computing devices within a hosted computing environment. Moreover, whileFIG. 2is described with respect to a pool manager119, the software within the memory210may additionally or alternatively include instructions to implement other components of the present disclosure, such as the content encoders113.

With reference toFIGS. 3A through 3C, a set of illustrative interactions for demand-based management of an encoder pool112and packager pool112will be described. As will be described below, the interactions ofFIGS. 3A through 3Cenable the number and configuration of devices within the encoder pool112and packager pool112to be altered based on a monitored demand for an output content stream. Illustratively, the various encoders114of the encoder pool112may provide enable either redundant or distributed content encoding within the content streaming system110. For example, a first portion of the encoders114may be configured to encode content according to a first set of parameters (e.g., at 4K resolution and 60 frames per second), while a second portion of the encoders114are configured to encode the same content according to a second set of parameters (e.g., at 1080p resolution and 30 frames per second). In some embodiments, an individual encoder114may be configured to encode multiple versions (or “renditions”) of content. For example, the second portion of the encoders114may, in addition to encoding the content according to the second set of parameters, also be configured to each encode the content according to a third set of parameters (e.g., at 480i resolution and 30 frames per second). Thus, any combination of encoders may be used to encode any combination of versions of a content item, with each encoder providing one or more redundant or cooperative encoded content streams.

In such an arrangement, it is sometimes desirable within the content streaming system110to configure the encoder pool112and the packager pool118to provide some level of redundancy, such that failure of one or more devices in a respective pool causes little or no perceptible failure in an output content stream to end users. However, increased redundancy is generally associated with increased computing resource usage (and thus, increased costs, environmental impact, wear and tear, etc.). Accordingly, the interactions ofFIGS. 3A through 3Ccan enable the redundancy of the pools112and118to vary according to the demand for the output content stream (or versions thereof), as can be monitored via the content distribution systems120, the content output devices102, or at the content streaming system110itself. In one embodiment, the redundancy of a pool can be measured according to a ratio of “hot” devices (those actively encoding content) to “warm” devices (those prepared to encode content, but not yet actively encoding content), as well as according to the total number of devices in a pool (as distributed according to the ratio). Further, the interactions ofFIG. 3A through 3Ccan enable other alterations to the configuration of pools112and118, such as changes in the locations of various encoders112or packagers116within the pools112and118. Illustratively, where demand for a content stream is high in a first geographic region, but low in a second geographic region, the configuration of the pools112and118may be altered such that the devices of the pools112and118are divided in location between the first and second region based on demand in those regions. As a further illustration, where devices of the pools112and118are located in a first geographic region with low demand for a content stream, but a second geographic region has relatively high demand for the content stream, the devices of the pools112and118may be least partially transferred to the second geographic region. Additionally or alternatively, devices within the first geographic region may be shut down, while new devices in the second geographic region are started.

In instances where demand is used to modify a number of devices within either or both pools112and118, the content streaming system110may a designate different number or ratio of devices for a number of “tiers” of demand, which can be defined according to an absolute number of end users with active sessions for viewing or outputting content. Illustratively, content streams with under ten thousand end users may be categorized as low demand, content streams with between ten thousand and one hundred thousand end users may be categorized as medium demand, content streams with between one hundred thousand and one million end users may be categorized as high demand, and content streams with over one million end users may be categorized as extreme demand. In one embodiment, higher tiers of demand may be associated with a higher absolute number of devices, as well as a higher proportion of “warmed” devices to “hot” devices. In this manner, failures of devices under higher demand are less likely to impact an output content stream, because output of the failed device can be recovered using interchangeable output of another device, and the failed device can be quickly replaced with a warmed device. Conversely, failures of devices under lower demand may be more likely to cause substantial impact to the content stream; however, this risk may be offset by the relatively low impact to end users overall as well as the reduced computing resource usage in the content streaming system110. In some embodiments, rather than using tiers of demand, a number or ratio of devices in a pool may be directly proportional to a number of end users (e.g., one hot device for every ten thousand users, one warm device for every hundred thousand users), or may correspond to the number of end users according to an algorithm defined by the content streaming system110or a content provider104.

The interactions ofFIG. 3Abegin at (1), where a content provider104transmits a request to the content streaming system110to encode content. The request may be transmitted, for example, via an application programming interface (API) or user interface (e.g., a graphical user interface, or GUI) provided by the content streaming system110. In some instances, the request may include information designating how end user demand for a content stream should be measured (e.g., tier definitions), as well as information designating how measurements of end user demand should be correlated to the number or ratio of devices in a pool. In other instances, this information may be provided by the content provider104separately, or may be provided by the content streaming system110(e.g., using default values). In some instances, the request may also include an expected demand for a content stream, which may be utilized to determine an initial configuration of pools.

At (2), the pool manager119determines the initial configuration for the encoder pool112and packager pool118. In one embodiment, the pool manager119can utilize the information from the request, such as the expected demand and the information for mapping expected demand to a number and configuration of instances, to determine the number and configuration of instances in the encoder pool112and packager pool118. In other instances, the pool manager119may determine the initial number and configuration of instances in the encoder pool112and packager pool118according to default values (e.g., with respect to either or both expected demand and correlation of that demand to resiliency of the pool). In still other instances, the pool manager119may determine the initial number and configuration of instances in the encoder pool112and packager pool118according to predicted values, such as based on prior demand data for similar content streams (e.g., streams within the same series of content, streams by the same author, director, producer, network, etc.).

Thereafter, at (3′) and (3″), the pool manager119transmits initialization instructions to the encoder pool112and the packager pool118, respectively. These instructions may include, for example, a number of devices to be included in the respective pools, and whether such devices should be configured as hot or warm. In some instances, the instructions further include a desired location or locations of the devices within the pools112and118. In other instances, locations of the pools112and118may be determined, for example, by transmission of the instructions to pools112and118within the desired location (e.g., where different pools112and118are implemented in various regions). Additionally, at (3′″), the pool manager119transmits pool information to the content ingestor111. This information may include, for example, the location (e.g., network address or other identifier) of the pool112, or devices within the pools112, such that the content ingestor111can transmit content to the pools112for encoding and packaging. The pool manager119can similarly notify the encoder pool112(e.g., within the initialization instructions) of a location or identifier of the packager pool118. At (4′) and (4″), the respective pools are initialized in accordance with the instructions. Illustratively, one or more host devices (not shown inFIG. 3A) may obtain the instructions from the pool manager119, and initiate execution of virtual machines executing an operating system and software to encode or package a content stream.

With reference toFIG. 3B, interactions for utilizing an encoder pool112and packager pool118to encode a content stream and distribute the stream to end users via content distribution systems120or content output devices102will be described. Illustratively, the interactions ofFIG. 3Bmay occur subsequently to those described above with respect toFIG. 3A, utilizing a previously initialized encoder pool112and packager pool118.

The interactions ofFIG. 3Bbegin at (1), where the content ingestor111receives content from a content provider104. Illustratively, the content ingestor111may receive content from content providers104over the network106(e.g., via a user datagram protocol, or “UDP,” stream). In other embodiments, the content ingest111may receive content directly from a capture device (e.g., a digital video recorder connected to the content ingestor111via a serial digital interface (“SDI”)). As noted above, the content provided to the content ingestor111may be in “raw,” uncompressed or analog format, or in a format that is otherwise required to be “encoded” prior to delivery to the content output devices102. Thus, at (2), the content ingestor111passes the content as a stream to the encoders114of the encoder pool112, which begin encoding content at (3). Illustratively, the encoders114may encode the obtained content into any number of known formats, including but not limited to H.263, H.264, H.265, MICROSOFT SMPTE 421M (also known as VC-1), APPLE™ ProRes, APPLE Intermediate Codec,VP3 through 9, Motion JPEG (“M-JPEG”), MPEG-2 part 2, RealVideo, Dirac, Theora, and MPEG-4 Part 2 (for video), and Vorbis, Opus, MP3, advanced audio coding (“AAC”), pulse-code modulation (“PCM”), dedicated to sound (“DTS”), MPEG-1, audio coding 3 (“AC-3”), free lossless audio codec (“FLAC”), and RealAudio (for audio), or combinations thereof. Various techniques for encoding content are known within the art, and therefore will not be described in greater detail herein. While it is assumed for the purposes of description ofFIG. 3Bthat each of the encoders114begin encoding the content stream at the same time, this is not required. Rather, where an encoder114begins encoding at a later point in time than other encoders114, that encoder114may establish synchronization with the other encoders114, such as via the mechanisms described in the '347 application.

After encoding content, the encoded content is transmitted to the packager pool118at (4). Thereafter, at (5), the content packagers116within the packager pool118may package the content into a container format accepted by the content distribution systems120and/or content output devices102. As will be recognized by one of skill in the art, a container format may generally combine encoded audio and video into a file, potentially along with synchronization information for the audio and video, subtitles, metadata, or other information. Examples of containers include, but are not limited to, Matroska, FLV, MPEG-4 Part 12, VOB, Ogg, Audio Video Interleave (“AVI”), Quicktime, Advanced Systems Format (“ASF”), RealMedia, and MPEG Transport Stream (“MPEG-TS”). In some instances, containers may include exclusively audio or exclusively video, rather than a combination of the two. In one embodiment, content packagers116may package the content into multiple container files, such as multiple transport stream files, and generate additional information enabling distribution systems120and/or content output devices102to distribute or consume the packaged content. For example, the content packagers116may package the encoded content according to streaming protocols, such as Hypertext Transport Protocol (“HTTP”) Live Streaming (“HLS”) or MPEG Dynamic Adaptive Streaming over HTTP (“MPEG-DASH”), and generate metadata regarding the encoded content, such as manifest file identifying each package of encoded content.

After packaging the encoded content, the content packagers116, at (6), distribute the content to one or more of the content output devices102and the content distribution systems120(which may in turn distribute the content to content output devices102for decoding and output) for display or output to end users.

With reference toFIG. 3C, interactions for modifying the number or configuration of devices within an encoder pool112and packager pool118based on demand from end users will be described. Illustratively, the interactions ofFIG. 3Cmay occur subsequently to those described above with respect toFIG. 3B, to modify the number or configuration of device within a previously initialized encoder pool112and packager pool118.

The interactions ofFIG. 3Cbegin at (1), where the content distribution system120distributes encoded and packaged content to content output devices102. Thereafter, either or both of the content distribution systems120and the content output devices102monitor consumption of the content, at (2′) and (2″), respectively. Illustratively, the content distribution system120may monitor consumption of content by determining a number of unique content output devices102to which content is transmitted within a threshold period of time. For example, the content distribution system120may monitor consumption of content by determining the number of network addresses (e.g., internet protocol or IP addresses) to which content is transmitted, or by determining a number of user accounts associated with the content output devices102to which content is transmitted. Additionally or alternatively, the content output devices102may monitor output of a content stream. For example, a media player on the content output device102may be configured to determine when a content stream is output by the device102, and transmit an indication of that output to the content distribution systems120or the pool manager119. In some instances, monitoring of content output on a content output device102may occur via computer-executable instructions provided in conjunction with the content stream, such as JAVASCRIPT™ instructions provided along with the content stream and executed on the content output device102. While logically grouped inFIG. 3C, the content output devices102may include devices that obtain content directly from the content streaming system110as well as those device that obtain content from the content distribution systems120. Moreover, it is contemplated not all content output devices102would necessarily be configured to monitor content consumption. However, at least a portion of such consumption may be monitored by the content distribution systems120. In some instances, the content streaming system110may be configured to adjust for unreported content consumption (e.g., by altering the monitored demand to account for expected unreported consumption, by altering the redundancy associated with demand based on monitored consumption, etc.).

Thereafter, at (3′) and (3″), the content distribution systems120and content output devices102report content consumption information to the pool manager119. The pool manager119, at (4), then updates the configuration of the encoder pool112and packager pool118based on the apparent popularity of the content stream (as reflected in consumption information) and a desired redundancy. While not shown inFIG. 3C, the pool manager119may in some instances be configured to deduplication the obtained content consumption information to adjust for duplication that may occur when an individual output of a content stream is reflected in both information from a content output device102and a content distribution system120. For example, the pool manager119may omit duplicative information indicating that the same user account or network address has output a content stream.

As noted above, different popularities of a content stream can be associated with different numbers and configurations of devices within either or both of the encoder pools112and packager pools118. For example, different popularities of a content stream may be described as “tiers” of popularity, based on a number of end users that output the stream. Each tier may be associated with a number of devices in a respective pool, as well as a ratio of hot to warm devices. As a further example, a number of hot or warm devices may be determined based on an algorithm linking numbers of end users with respective numbers of hot or warm devices. The correlations between end user demand and number or configuration of devices may be provided by a content provider104of the content stream or the content streaming system110. Accordingly, at (4), the pool manager119can update the configuration of the encoder pool112and the packager pool118as needed to adjust the number and configuration of devices within the respective pools to an appropriate value. Thereafter, at (5′), the pool manager119can transmit updated configuration information (if necessary) to either or both of the encoder pool112and packager pool118. The respective pools112and118can then modify the number or configuration of devices within the pools112and118. Such modifications may include, for example, deconstructing or shutting down one or more virtual machines within the pools, or causing a virtual machine to halt processing of a content stream while maintaining a ready (warmed) state to resume processing. Such modifications may further include adding devices into the pool112, such as by loading a new virtual machine to processes a content stream or maintain a ready (warmed) state. Further, such modifications may include causing previously warmed devices to begin encoding or packaging a content stream. Still further, such modifications may include varying configurations of individual devices within the pools112and118, such as by varying an individual virtual machine instances access to computing resources of a host device. In some instances, such modifications may include altering a location of devices within the pools112and118(e.g., by migrating the devices or by shutting down devices in a first location and booting devices in a second location). In some instances, modifications to an encoder pool112may require synchronization of the encoders114within the pool, to ensure that output content is seamlessly interchangeable. The encoders114may therefore utilize a synchronization protocol, such as that described in the '347 application, to maintain synchronization within the encoder pool112. In addition, at (5″), the pool manager119can transmit updated configuration information to the content ingestor111, as necessary, in order to inform the content ingestor111of any changes within the encoder pool112. For example, the pool manager119may transmit updated network address information for devices within the pool112to the content ingestor111, in order to enable the content ingestor111to transmit content for encoding to the devices. Thus, the number and configuration of devices in the encoder pool112and the packager pool118can be dynamically adjusted according to a monitored consumption by end users.

While the interactions ofFIGS. 3A through 3Care described above sequentially, it is contemplated that some or all of the interactions may repeatedly occur, and that these interactions may occur at least in part concurrently. For example, the interactions ofFIG. 3Cmay be followed by those ofFIG. 3B, and this sequence may be repeated while content is streamed to content output devices102. Thus, the interactions ofFIG. 3A through 3Ccan implement a continuous feedback loop, dynamically adjusting the size and configuration of the encoder and packager pools112and118according to end user demand, thereby adjusting the resiliency of the respective pools112and118in accordance with their popularity, while reducing the inefficiencies caused by implementing excessive resiliency for low popularity streams. Moreover, while the interactions ofFIGS. 3A through 3Care described with respect to illustrative examples of how encoder pools112or packager pools118may be modified according to various criteria, additional or alternative criteria are contemplated within the present disclosure. For example, while the number, location, or configuration of devices within an encoder pool112or packager pool118may be varied based on measured demand, the devices with such pools112and118may additionally or alternatively be varied based on predicted demand (e.g., due to peak viewing times in a current time zone). With reference toFIG. 4, one illustrative routine400is depicted that may be implemented to manage a size and configuration of encoder pools or packager pools based on end user demand. The routine400may be implemented, for example, by the pool manager119ofFIG. 1. The routine400begins at block402, where the pool manager119obtains an encoding request from a content provider104. In some instances, the encoding request may include information for determining a desired number and configuration of encoding devices based on end user demand, such as information designating “tiers” of popularity and desired pool configurations corresponding to the designated tiers. In another instance, the encoding request may request to utilize default values for determining a desired number and configuration of encoding devices based on end user demand. As noted above, in some instances the pool manager119may be configured to maintain a static or near-static number of devices within a pool or across pools, while altering the specific content versions output by such devices (e.g., low definition, standard definition, high definition) proportionally to the demand for those versions. Accordingly, a request may include information for proportioning devices between versions based on demand, or request may implicate default values or algorithms for proportioning devices between versions.

At block404, the pool manager119determines an initial configuration of encoding pools or packager pools based on the request. Illustratively, the initial configuration may be based on a default anticipated demand level, a demand level specified by a content provider104, or a demand level determined based on historical information. The initial configuration may include, for example, a number of devices to include within either or both an encoder pool and packager pool, as well as whether each device should be operated as hot (encoding content) or warm (ready to encode content). The initial configuration may further proportion such devices between different versions of a content, or establish different pools for different versions.

At block406, the pool manager119can transmit instructions to the encoder pool and packager pool (or to one or more devices hosting virtual machines implementing the respective pools) to initialize the pools according to the determined configuration. Thereafter, at block408, the encoder pool and packager pool can begin encoding content. In some instances, the encoder pool may operate to synchronize encoding, such that output content streams as seamlessly interchangeable and the encoder pool operates in a redundant manner. Embodiments for creation of seamlessly interchanging are described in the '347 application.

At block410, the pool manager119obtains consumption data from either or both content output devices102and content distribution systems120. While not shown inFIG. 4, the pool manager119may process the obtained consumption data to more accurately reflect end user consumption, such as by deduplicating the consumption data or adjusting the consumption data to reflect an expected underreporting of content output by end users.

At block412, the pool manager119determines whether the configuration of either or both the encoder pool or the packager pool should be adjusted based on the consumption data, as well as based on information correlating consumption data to an appropriate number or configuration of devices in the encoder pool or packager pool. Illustratively, where consumption data indicates a higher than expected demand or popularity of a content stream, the pool manager119can determine, at block412, that the number of devices within either or both of the encoder or packager pool should be increased, that previously warmed devices should begin encoding content, or that the computing resources available to individual devices within the encoder or packager pool should be increased. Conversely, where consumption data indicates a lower than expected demand or popularity of a content stream, the pool manager119can determine, at block412, that the number of devices within either or both of the encoder or packager pool should be decreased, or that the computing resources available to individual devices within the encoder or packager pool should be decreased. As a further illustration, if demand for a first content version is higher than expected while demand for a second content version is lower than expected, the pool manager119may determine, at block412, that devices previously configured to generate the first content version should be reconfigured to generate the second content version. As yet another illustration, if the demand for a content (or content version) among different regions or locations varies, the configuration of the pools112and118may be altered to address that variation. In some instances, implementation of block412may include consideration of data in addition or alternatively to consumption data, such as expected demand levels for a content based on a current time (e.g., predicted demand during peak viewing times, predicted demand during low viewing times, etc.).

In the instance that no reconfiguration or altering of the encoder pool or packager pool is required, the routine400can proceed to block416. However, if reconfiguration of either the encoder pool or packager pool is required, the routine400proceeds to block414, where the pool manager119determines an updated configuration of either or both the encoder pool or the packager pool based on the consumption data, as well as the information correlating consumption data to an appropriate number or configuration of devices in the encoder pool or packager pool. As noted above, an updated configuration may further be based on additional or alternative data, such as expected demand levels for a content based on a current time. The routine400then returns to block406, where the pool manager119transmits instructions to the encoder pool or packager pool to configure the pools according to the updated configuration.

The routine400then proceeds as described above until reaching block416, where implementation of the routine400varies according to whether a content stream is still being encoded and provided to end users. While the content stream is being encoded, block416evaluates as true, and the routine400returns to block408, where it proceeds as described above. When content is no longer encoded, the routine400ends at block418.

All of the methods and processes described above may be embodied in, and fully automated via, software code modules executed by one or more computers or processors. The code modules may be stored in any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods may alternatively be embodied in specialized computer hardware.

Unless otherwise explicitly stated, articles such as ‘a’ or ‘an’ should generally be interpreted to include one or more described items. Accordingly, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices can also be collectively configured to carry out the stated recitations. For example, “a processor configured to carry out recitations A, B and C” can include a first processor configured to carry out recitation A working in conjunction with a second processor configured to carry out recitations B and C.