Patent Description:
To meet the demand for wireless data traffic having increased since deployment of <NUM>th generation (<NUM>) communication systems, efforts have been made to develop an improved <NUM>th generation (<NUM>) or pre-<NUM> communication system. Therefore, the <NUM> or pre-<NUM> communication system is also called a `Beyond <NUM> Network' or a 'Post Long Term Evolution (LTE) System'.

The <NUM> communication system is considered to be implemented in higher frequency (mmWave) bands, e.g., <NUM> or <NUM> bands, so as to accomplish higher data rates.

In the <NUM> system, Hybrid frequency shift keying (FSK) and quadrature amplitude modulation (FQAM) and sliding window superposition coding (SWSC) as an advanced coding modulation (ACM), and filter bank multi carrier (FBMC), non-orthogonal multiple access (NOMA), and sparse code multiple access (SCMA) as an advanced access technology have been developed.

In the <NUM> system, various communication schemes are discussed. For example, a grant-free communication scheme for transmitting data without granting an uplink transmission is proposed. Furthermore, various discussions for supporting the grant-free communication efficiently are underway.

"<NPL>) discloses Network Based Media Processing (NBMP) in relation to Coded representation of immersive media (MPEG-<NUM>). The NBMP framework enables creators, service providers and consumers of digital media to describe media processing operations that are to be performed by media processing entities connected through digital network described therein, and provides a method to describe workflows of a media processing service as a composition of a set of processing functions provided by a media processing entity which are accessible though NBMP APIs.

<CIT> discloses a method including receiving a first Internet protocol (IP) flow for an IP session for a subscriber; selecting a first service function group from a plurality of service function groups to perform one or more services for the IP session for the subscriber, wherein each of the plurality of service function groups comprises a plurality of service function chain types and wherein each service function chain type comprises an ordered combination of one or more service functions; assigning the IP session for the subscriber to the first service function group; and forwarding the first IP flow for the IP session of the subscriber across a first service function chain type for the first service function group based, at least in part, on a service policy for the subscriber.

Further details are defined in the dependent claims.

This disclosure provides method and apparatus for management of network based media processing functions.

According to various embodiments of the present disclosure, a method for operating an electronic device is provided according to appended claim <NUM>.

According to various embodiments of the present disclosure, an electronic device is provided according to appended claim <NUM>.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts:.

Cloud media processing is gaining traction where media processing workloads are setup in the network (e.g., cloud) to take advantage of advantages of the benefits offered by the cloud such as (theoretically) infinite compute capacity, auto-scaling based on need, and on-demand processing. An end user client can request a network media processing provider for provisioning and configuration of media processing functions as required. The provider typically provides a list of processing functions/services that are available in the provider's domain that the end user clients can request provisioning as part of the processing workflow in the network. To provide a facility for lookup of supported media processing functions, the provider has to implement a function repository that lists all the supported media processing functions.

Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably-arranged system or device.

An electronic device, according to embodiments of this disclosure, can include a personal computer (such as a laptop or a desktop), a workstation, a server, a television, an appliance, a virtual assistant, and the like. Additionally, the electronic device can be at least one part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or a measurement device. In some embodiments, the electronic device can be a portable electronic device like a portable communication device (such as a smartphone or mobile phone), a laptop, a tablet, an electronic book reader (such as an e-reader), a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a virtual reality headset, a portable game console, a camera, or a wearable device, among others. The electronic device can be one or a combination of the above-listed devices. Additionally, the electronic device as disclosed herein is not limited to the above-listed devices and can include new electronic devices depending on the development of technology. It should be noted that, as used here, the term "user" may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.

<FIG> illustrates an example communication system <NUM> in accordance with an embodiment of this disclosure. The embodiment of the communication system <NUM> shown in <FIG> is for illustration only. Other embodiments of the communication system <NUM> can be used without departing from the scope of this disclosure.

The communication system <NUM> includes a network <NUM> that facilitates communication between various components in the communication system <NUM>. For example, the network <NUM> can communicate Internet Protocol (IP) packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, or other information between network addresses. The network <NUM> includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems at one or more locations.

In this example, the network <NUM> facilitates communications between a server <NUM> and various client devices <NUM>-<NUM>. The client devices <NUM>-<NUM> may be, for example, a smartphone, a tablet computer, a laptop, a personal computer, a wearable device, a head-mounted display (HMD), or the like. The server <NUM> can represent one or more servers. Each server <NUM> includes any suitable computing or processing device that can provide computing services for one or more client devices, such as the client devices <NUM>-<NUM>. Each server <NUM> could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network <NUM>. In some embodiments, the server <NUM> includes a workflow manager that can select functions and build a workflow pipeline to perform a media processing task. The workflow manager is discussed in greater detail below with respect to <FIG>, and <FIG>.

Each client device <NUM>-<NUM> represents any suitable computing or processing device that interacts with at least one server (such as the server <NUM>) or other computing device(s) over the network <NUM>. The client devices <NUM>-<NUM> include a desktop computer <NUM>, a mobile telephone or mobile device <NUM> (such as a smartphone), a PDA <NUM>, a laptop computer <NUM>, and a tablet computer <NUM>. However, any other or additional client devices could be used in the communication system <NUM>. Smartphones represent a class of mobile devices <NUM> that are handheld devices with mobile operating systems and integrated mobile broadband cellular network connections for voice, short message service (SMS), and Internet data communications.

In this example, some client devices <NUM>-<NUM> communicate indirectly with the network <NUM>. For example, the client devices <NUM> and <NUM> (mobile device <NUM> and PDA <NUM>, respectively) communicate via one or more base stations <NUM>, such as cellular base stations or eNodeBs (eNBs). Also, the client devices <NUM> and <NUM> (laptop computer <NUM> and tablet computer <NUM>, respectively) communicate via one or more wireless access points <NUM>, such as IEEE <NUM> wireless access points. Note that these are for illustration only and that each client device <NUM>-<NUM> could communicate directly with the network <NUM> or indirectly with the network <NUM> via any suitable intermediate device(s) or network(s).

In some embodiments, any of the client devices <NUM>-<NUM> transmits information securely and efficiently to another device, such as, for example, the server <NUM>. Also, any of the client devices <NUM>-<NUM> can trigger the information transmission between itself and server <NUM>.

Although <FIG> illustrates one example of a communication system <NUM>, various changes can be made to <FIG>. For example, the communication system <NUM> could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. While <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> and <FIG> illustrate example electronic devices in accordance with an embodiment of this disclosure. In particular, <FIG> illustrates an example server <NUM>, and the server <NUM> could represent the server <NUM> in <FIG>. The server <NUM> can represent one or more local servers, one or more remote servers, clustered computers, and components that act as a single pool of seamless resources, a cloud-based server, and the like. The server <NUM> can be accessed by one or more of the client devices <NUM>-<NUM> of <FIG>.

As shown in <FIG>, the server <NUM> includes a bus system <NUM> that supports communication between at least one processing device <NUM>, at least one storage device <NUM>, at least one communications interface <NUM>, and at least one input/output (I/O) unit <NUM>. The processor <NUM> executes instructions that can be stored in a memory <NUM>. The processor <NUM> can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processors <NUM> include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The memory <NUM> and a persistent storage <NUM> are examples of storage devices <NUM> that represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, or other suitable information on a temporary or permanent basis). The memory <NUM> can represent a random access memory or any other suitable volatile or non-volatile storage device(s). The persistent storage <NUM> can contain one or more components or devices supporting longer-term storage of data, such as a read only memory, hard drive, Flash memory, or optical disc.

The communications interface <NUM> supports communications with other systems or devices. For example, the communications interface <NUM> could include a network interface card or a wireless transceiver facilitating communications over the network <NUM>. The communications interface <NUM> can support communications through any suitable physical or wireless communication link(s).

The I/O unit <NUM> allows for input and output of data. For example, the I/O unit <NUM> can provide a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input device. The I/O unit <NUM> can also send output to a display, printer, or other suitable output device. Note, however, that the I/O unit <NUM> can be omitted, such as when I/O interactions with the server <NUM> occur via a network connection.

Note that while <FIG> is described as representing the server <NUM> of <FIG>, the same or similar structure could be used in one or more of the various client devices <NUM>-<NUM>. For example, a desktop computer <NUM> or a laptop computer <NUM> could have the same or similar structure as that shown in <FIG>.

<FIG> illustrates an example electronic device <NUM>, and the electronic device <NUM> could represent one or more of the client devices <NUM>-<NUM> in <FIG>. For example, in some embodiments, the electronic device <NUM> may implement or represent a virtual assistant device that can receive a natural language input, derive meaning from the input, and perform an action based on the derived meaning. The electronic device <NUM> can be a mobile communication device, such as, for example, a mobile station, a subscriber station, a wireless terminal, a desktop computer (similar to desktop computer <NUM> of <FIG>), a portable electronic device (similar to the mobile device <NUM>, PDA <NUM>, laptop computer <NUM>, or tablet computer <NUM> of <FIG>), and the like.

As shown in <FIG>, the electronic device <NUM> includes an antenna <NUM>, a communication unit <NUM>, transmit (TX) processing circuitry <NUM>, a microphone <NUM>, and receive (RX) processing circuitry <NUM>. The communication unit <NUM> can include, for example, a radio frequency (RF) transceiver, a BLUETOOTH transceiver, a WI-FI transceiver, a ZIGBEE transceiver, an infrared transceiver, and the like. The electronic device <NUM> also includes a speaker <NUM>, a processor <NUM>, an input/output (I/O) interface (IF) <NUM>, an input <NUM>, a display <NUM>, a memory <NUM>, and a sensor(s) <NUM>. The memory <NUM> includes an operating system (OS) <NUM>, one or more applications <NUM>, and media content.

The communication unit <NUM> receives, from the antenna <NUM>, an incoming RF signal transmitted from an access point (such as a base station, WI-FI router, or BLUETOOTH device) or other device of the network <NUM> (such as a WI-FI, BLUETOOTH, cellular, <NUM>, LTE, LTE-A, WiMAX, or any other type of wireless network). The communication unit <NUM> down-converts the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuitry <NUM> that generates a processed baseband signal by filtering, decoding, and/or digitizing the baseband or intermediate frequency signal.

The TX processing circuitry <NUM> receives analog or digital voice data from the microphone <NUM> or other outgoing baseband data from the processor <NUM>. The outgoing baseband data can include web data, e-mail, or interactive video game data. The TX processing circuitry <NUM> encodes, multiplexes, and/or digitizes the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The communication unit <NUM> receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuitry <NUM> and up-converts the baseband or intermediate frequency signal to an RF signal that is transmitted via the antenna <NUM>.

The processor <NUM> can include one or more processors or other processing devices. The processor <NUM> can execute instructions that are stored in a memory <NUM>, such as the OS <NUM> in order to control the overall operation of the electronic device <NUM>. For example, the processor <NUM> could control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit <NUM>, the RX processing circuitry <NUM>, and the TX processing circuitry <NUM> in accordance with well-known principles. The processor <NUM> can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in some embodiments, the processor <NUM> includes at least one microprocessor or microcontroller. Example types of processor <NUM> include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The processor <NUM> is also capable of executing other processes and programs resident in the memory <NUM>, such as operations that receive, store, and timely instruct by providing ASR processing and the like. The processor <NUM> can move data into or out of the memory <NUM> as required by an executing process. In some embodiments, the processor <NUM> is configured to execute a plurality of applications <NUM> based on the OS <NUM> or in response to signals received from external source(s) or an operator. Example, applications <NUM> can include a camera application (for still images and videos), a video phone call application, an email client, a social media client, an SMS messaging client, a virtual assistant, and the like. In some embodiments, the processor <NUM> is configured to receive and transmit the media content <NUM>. The processor <NUM> is also coupled to the I/O interface <NUM> that provides the electronic device <NUM> with the ability to connect to other devices, such as client devices <NUM>-<NUM>. The I/O interface <NUM> is the communication path between these accessories and the processor <NUM>.

The processor <NUM> is also coupled to the input <NUM> and the display <NUM>. The operator of the electronic device <NUM> can use the input <NUM> to enter data or inputs into the electronic device <NUM>. The input <NUM> can be a keyboard, touchscreen, mouse, track ball, voice input, or other device capable of acting as a user interface to allow a user in interact with electronic device <NUM>. For example, the input <NUM> can include voice recognition processing, thereby allowing a user to input a voice command. In another example, the input <NUM> can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel can recognize, for example, a touch input in at least one scheme, such as a capacitive scheme, a pressure sensitive scheme, an infrared scheme, or an ultrasonic scheme. The input <NUM> can be associated with sensor(s) <NUM> and/or a camera by providing additional input to processor <NUM>. In some embodiments, the sensor <NUM> includes one or more inertial measurement units (IMUs) (such as accelerometers, gyroscope, and magnetometer), motion sensors, optical sensors, cameras, pressure sensors, heart rate sensors, altimeter, and the like. The input <NUM> can also include a control circuit. In the capacitive scheme, the input <NUM> can recognize touch or proximity.

The display <NUM> can be a liquid crystal display (LCD), light-emitting diode (LED) display, organic LED (OLED), active matrix OLED (AMOLED), or other display capable of rendering text and/or graphics, such as from websites, videos, games, images, and the like.

The memory <NUM> can include persistent storage (not shown) that represents any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information). The memory <NUM> can contain one or more components or devices supporting longer-term storage of data, such as a read only memory, hard drive, Flash memory, or optical disc. The memory <NUM> also can contain media content <NUM>. The media content <NUM> can include various types of media such as images, videos, three-dimensional content, VR content, AR content, and the like.

The electronic device <NUM> further includes one or more sensors <NUM> that can meter a physical quantity or detect an activation state of the electronic device <NUM> and convert metered or detected information into an electrical signal. For example, the sensor <NUM> can include one or more buttons for touch input, a camera, a gesture sensor, an IMU sensors (such as a gyroscope or gyro sensor and an accelerometer), an air pressure sensor, a magnetic sensor or magnetometer, a grip sensor, a proximity sensor, a color sensor, a bio-physical sensor, a temperature/humidity sensor, an illumination sensor, an Ultraviolet (UV) sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG) sensor, an Electrocardiogram (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, and the like. The sensor <NUM> can further include control circuits for controlling any of the sensors included therein. Any of these sensor(s) <NUM> can be located within the electronic device <NUM>.

Although <FIG> and <FIG> illustrate examples of electronic devices, various changes can be made to <FIG> and <FIG>. For example, various components in <FIG> and <FIG> could be combined, further subdivided, or omitted and additional components could be added according to particular needs. In addition, as with computing and communication, electronic devices and servers can come in a wide variety of configurations, and <FIG> and <FIG> do not limit this disclosure to any particular electronic device or server.

<FIG> illustrates a block diagram of an example network media processing system <NUM> in accordance with an embodiment of this disclosure. <FIG> illustrates and example workflow 436a in accordance with an embodiment of this disclosure. <FIG> illustrates an example media processing function <NUM> in accordance with an embodiment of this disclosure. <FIG> illustrates a function group <NUM> in accordance with an embodiment of this disclosure. The embodiments of <FIG> are for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

As shown in <FIG>, the network media processing system <NUM> includes a media source <NUM>, a server <NUM> and a media sink <NUM> in communication over a network <NUM>. The network <NUM> can be the same as or similar to the network <NUM> of <FIG>. In some embodiments, the network <NUM> represents a "cloud" of computers interconnected by one or more networks, where the network is a computing system utilizing clustered computers and components that act as a single pool of seamless resources when accessed. Also, in some embodiments, the network <NUM> is connected with one or more servers (such as the server <NUM> of <FIG>, the server <NUM>, and the server <NUM>), one or more electronic devices (such as the client devices <NUM>-<NUM> of <FIG>, the electronic device <NUM>, and the media source <NUM>). Further, in some embodiments, the network <NUM> can be connected to an information repository (such as the media sink <NUM>, and database), that contains a look-up tables and information pertaining to various functions as well as a repository of published media.

In some embodiments, the media source <NUM> and the media sink <NUM> can represent one of the client devices <NUM>-<NUM> of <FIG>, the electronic device <NUM> of <FIG>, or other suitable device. In other embodiments, a portion of the components included in the media source <NUM> and the media sink <NUM> can be included in different devices, such as the server <NUM>, multiple servers <NUM> or <NUM>, multiple client devices <NUM>-<NUM>, or other combination of different devices. In some embodiments, the media source <NUM> and the media sink <NUM> are the same device.

In this example, the media source <NUM> includes an information repository <NUM>. Similarly the media sink <NUM> can includes an information repository <NUM>. The media source <NUM> can include a camera or additional components that can capture or receive media. In some embodiments, the captured or recorded media requires a certain type of processing such as VR stitching, but lacks the processing capabilities to perform the necessary processing of the media content. The media sink <NUM> represents a storage device that the processed media can be delivered after processing by the server <NUM>.

The information repository <NUM> and <NUM> represent any suitable structure(s) capable of storing and facilitating retrieval of information (such as data, program code, or other suitable information on a temporary or permanent basis). The information repository <NUM> and <NUM> can include a memory and a persistent storage. The memory can be RAM or any other suitable volatile or non-volatile storage device(s), while the persistent storage can contain one or more components or devices supporting longer-term storage of data, such as a ROM, hard drive, Flash memory, or optical disc. The information repositories <NUM> and <NUM> can include one or more media content such as the media content <NUM> of <FIG>.

The media source <NUM> can include a user interface that enables a user to select media content to be offloaded from the information repository <NUM> to the server <NUM> for processing. The user interface can also enable the user to provide instructions to the server <NUM> as to what type of media processing is to be performed with respect to the media content. The media source <NUM> can offload media content for processing on the server <NUM>. The server <NUM> can perform a network based media processing workflow by creating as a workflow <NUM> of media processing functions (or tasks) for a received media processing request. For example, a media source, such as the media source <NUM>, sends a workflow description to the server <NUM>. The workflow description provides a description of the input. For example, if the input is based on recordings from multiple cameras, the input description can include the number of cameras that recorded the content, the relative position of the cameras with respect to each other, the format the content was captured, and the like. The workflow description can also include a request for the intended media processing. Additionally, the workflow description can include a location (such as the media sink <NUM>) the media content is to be moved to after the processing, such as an expected distribution output post processing.

After the server <NUM> receives the workflow description from the media source <NUM>, the server <NUM> identifies one or more functions, entities, tasks, services, and the like to perform the media processing based on the workflow description and information associated with each of the functions.

In some embodiments, the server <NUM> can be implemented as shown in <FIG>. In other embodiments, a portion of the components included in the server <NUM> can be included in different devices, such as multiple servers <NUM> or <NUM>, multiple client devices <NUM>-<NUM>, multiple electronic devices <NUM>, or a combination of different devices. The server <NUM> can represent one or more local servers, one or more remote servers, a network based media processing (NBMP) framework, or the like. The server <NUM> can be a web server, a server computer such as a management server, or any other electronic computing system capable of sending and receiving data. The server <NUM> can include a function repository <NUM>, a workflow manager <NUM>, and a workflow <NUM>.

The function repository <NUM> can be the same as or similar to the information repositories <NUM> and <NUM>. The function repository <NUM> represents any suitable structure(s) capable of storing and facilitating retrieval of functions. The function repository <NUM> can include a memory and a persistent storage. The memory can be RAM or any other suitable volatile or non-volatile storage device(s), while the persistent storage can contain one or more components or devices supporting longer-term storage of data, such as a ROM, hard drive, Flash memory, or optical disc.

The function repository <NUM> can include multiple network based media processing functions, such as the media processing function <NUM> (the function <NUM>) of <FIG>. In some embodiments, network based media processing functions, such as the function <NUM>, are located on a remote server. The function <NUM> represents a network based media processing function, a network based media processing task, a network based media processing service, a network based media processing entity or the like. The function <NUM> is a task or node of the workflow 436a of <FIG>.

The function <NUM> can implement one media processing function or task. When multiple functions are arranged in a workflow pipeline, with each function performing a certain processing task, a workflow such as the workflow <NUM> and the workflow 436a of <FIG> are created. Each of the network based media processing functions when included in a workflow pipeline performs their corresponding tasks in order to complete the requested task from the media source <NUM>. Each function <NUM> in the function repository <NUM> can be defined using a set of descriptors. Examples of descriptors for defining a function <NUM> are shown in the following table.

All media processing functions, such as the function <NUM>, are configured individually by the workflow manager <NUM>. In order for the function <NUM> to perform a task within a workflow, the function <NUM> receives requirements <NUM>, input media (or metadata or both) <NUM>, and configuration information <NUM>. The function <NUM> then produces an output <NUM> based on the received inputs (requirements <NUM>, input data <NUM>, and the configuration information <NUM>).

A media processing function or task (such as the function <NUM>) can receive an input data <NUM>, such as a media data streams, metadata, or both. The function <NUM> can process the received media data streams and metadata. The function <NUM> produces the output <NUM> such as an output media, metadata, or both. The server <NUM> can use multiple media processing functions of different types to perform the requested processing of the media content. In some embodiments, the network based media processing functions can be created by a third party service provider and included in the directory.

The function repository <NUM> can also include a directory or list of functions including network based media processing functions, network based media processing tasks, network based media processing services, and the like, which are available to the workflow manager <NUM>. The list of functions can also be located remotely from the server <NUM>, such as in a remote database. The directory includes details of each media processing function accessible to the workflow manager <NUM>. The directory can list the details concerning each of the media processing functions, such as (i) task details, (ii) inputs (such as the requirements <NUM>, input data <NUM>, and the configuration information <NUM> of the function <NUM> of <FIG>), and (iii) outputs (such as the output <NUM> of the function <NUM> of <FIG>).

As shown in <FIG>, the workflow description, which provides information for media processing, is sent from the NBMP source <NUM> to the workflow manager <NUM>, which the workflow manager <NUM> uses to select a set of MPEs and run media processing tasks in them. The tasks to be run in the MPE are chosen from a function repository <NUM>. The workflow manager <NUM> then runs the selected tasks in one or more media processing entities; configures the tasks; and connects control and data plane paths between the NBMP source <NUM>, the provisioned MPEs, and the media sink <NUM>. The workflow can be shown as a directed acyclic graph (DAG) as shown in <FIG>.

The "input data" is represented as the input data <NUM> to the function <NUM>. The input data is the media content, metadata, or a portion of the media content that the particular function is to act on. For example, "input data" can specify a format type of the data, such as a specific format of the data. The type of data can be the actual input data stream or a metadata stream. That is, the input data represents the data that the function uses or manipulates. In some embodiments, the input data can be an input description that provides information describing the type of input for the particular function.

The "configuration data" is represented as the configuration information <NUM> of the function <NUM> of <FIG>. The "configuration data" provides configuration information to the function so that the function can initialize its processing parameters with respect to the "input data. " For example, the "configuration data" can provide information that is needed for executing the assigned processing for the function. The "configuration data" can include configuration variables, constants, and parameters required by the executable/script assigned to the particular function. The "configuration data" can also include the configuration type of output that the function is to create. The "configuration data" can also include the number of cameras and orientation of each camera when recording the media.

The "configuration data" is represented as the configuration information <NUM> of the function <NUM> of <FIG>. The "configuration data" provides configuration information to the function so that the function can initialize its processing parameters with respect to the "input data. " For example, the "configuration data" can provide information that is needed for executing the assigned processing for the function. The "configuration data" can include configuration variables, constants, and parameters required by the executable/script assigned to the particular function. The "configuration data" can also include the configuration type of output that the function is to create. The "configuration data" can also include the number of cameras and orientation of each camera when recording the medial. For example, if the recorded media is captured by four cameras, the "configuration data" can include the orientation of each camera such as the location of each camera relative to the other three cameras, the configuration parameters of the cameras, such as the depth of focus, the number of megapixels, lighting conditions, resolution, and the like.

The "requirements" is represented as the requirements <NUM> of the function <NUM> of <FIG>. The "requirements" provides one or more requirements and pre-condition information to determine where the function should be located and how it is selected by the workflow manager <NUM>. For example, the "requirements" can include QoS information such as delay requirements such as delay requirements, bandwidth requirements, and the like. The "requirements" can also include processing requirements such as compute requirements, storage requirements, infrastructure requirements, and the like that are needed by the particular function in order to perform the required processing. For instance, the processing requirements can indicate particular hardware that the function uses such as a particular processor, such as the number of processors required, by the function the speed of the processors required by the function, the minimum or maximum memory requirements required by the function, the delay or latency for the function to perform its task, and the like.

The workflow manager <NUM> can receive the workflow description from the media source <NUM>. The workflow manager <NUM> can search through all of the available functions and services within the directory (such as the directory within the function repository <NUM>). Based on the information of each function that is included in the directory, the workflow manager <NUM> inspects requirements of the functions and elects one or more network based media processing functions to build the workflow <NUM>. An example workflow 436a is illustrated below in <FIG>. The workflow manager <NUM> selects and maps each of the media processing functions to create a pipeline the workflow <NUM>.

For example, once the network operator or third party service provider defines the multiple functions that are represented in the directory, the server <NUM> can receive a request for media processing. The workflow manager <NUM> can receive a media processing request from the media source <NUM>. The request for media processing can include a particular input of media data, a requested media output of the processing as well as certain processing requirements such as delays and the like. In some embodiments, the media source <NUM> represents multiple electronic devices, each of which can sent media processing requests to the server <NUM>.

When a request for media processing is received from the media source <NUM>, the workflow manager <NUM> scans the directory that includes all of the available services and functions. In response to receiving the request, the workflow manager <NUM> inspects the specified requirements for each function, service, task, and the like that are included in the directory. For example, based on the request and functions within the directory, the workflow manager <NUM> reviews each functions details, inputs, output (see inputs of Table (<NUM>) above) when selecting each function.

The workflow manager <NUM> selects certain functions from the directory which are able to perform the processing. The workflow manager <NUM> uses selected functions to build the media processing pipeline, such as the workflow <NUM>. For example, the workflow manager <NUM> can select each subsequent function of the workflow <NUM> based on the output of the previous function, until the requested end result is able to be accomplished. Such that the output of a first function is the input of a second function, the output of the second function is the input of a third function, and so-on until the requested processing is complete. In some embodiments, the workflow <NUM> is not linear such as the workflow 436a as illustrated in <FIG>. In some embodiments, the workflow manager <NUM> can instruct certain functions to produce multiple outputs such that the output is sent to a corresponding number of subsequent functions. The workflow manager <NUM> also selects particular functions based on the processing requirement included in the request. Thereafter, the workflow manager <NUM> organizes each selected function in a pipeline, such as the workflow <NUM> to perform the media processing.

The workflow manager <NUM> can also selects certain functions based on the overall latency of each individual function. For example, if the workflow is to be complete within a certain time duration (as indicated in the received workflow description), the workflow manager <NUM> selects certain functions that can perform the entire workflow within the allotted end-to-end latency as indicated in the received workflow description. The workflow manager <NUM> can also select certain functions based on a maximum or minim allowed frame rate, process speed, and the like.

As detailed above each function includes certain requirements in order to execute appropriately. The requirements can include a number of CPUs, a number of GPUs, a memory requirement, a bandwidth requirement and the like. When the workflow manager <NUM> selects certain functions, the workflow manager <NUM> inspects the requirements of each function and selects functions based on the system parameters to ensure that each function has the necessary requirements to perform its respective processing task.

The workflow manager <NUM> maps the source request to appropriate media processing functions in the workflow <NUM> based on each functions pre-defined requirements. The workflow manager <NUM> then maps the source request to each function in the workflow <NUM> based on the functions services defined the in the requirements of each function (as shown in Table (<NUM>) above).

The workflow manager <NUM> can monitor each individual function as each function performs its processing of the media. Monitoring each function can include identifying if a function fails while performing its task. Monitoring each function can also include identifying if the function can requires a different input format than indicated in the directory or produce a different output than indicated in the directory. If the workflow manager <NUM> identifies that a function fails, the workflow manager <NUM> can select one or more new functions to replace the failed function in the workflow <NUM>. Similarly, if the workflow manager <NUM> identifies that the input or output of a function is incorrect to perform the intended workflow, then the workflow manager <NUM> can remove the incorrect function, and replace it with one or more new functions. Alternatively, the workflow manager <NUM> can select one or more additional functions from the directory that changes the format of the input or output of a function in order that the workflow of the workflow <NUM> processes the media content.

The workflow <NUM> represents a workflow pipeline with any number of selected functions that are mapped in a certain order to perform the processing request. In some embodiments, the workflow <NUM> can be linear workflow such that the output of each function is the input to the next subsequent function, until the processing is complete. In some embodiments, the workflow <NUM> represents a workflow that is not linear as shown in in the workflow 436a of <FIG>. In some embodiments, the functions in a workflow pipeline can occur in series, while other functions occur in parallel (at the same or similar time).

In some embodiments, an end user can select and organize the individual functions into a workflow pipeline and allow customers the customers of the end user to use the created function. In some embodiments, a user can access the workflow manager <NUM> and request the workflow manager <NUM> to create a workflow pipeline based on the requested processing task and input data.

<FIG> illustrates an example function group <NUM> in accordance with an embodiment of the present disclosure. The set of media processing functions <NUM> that can be inserted into media workflows can be managed inside a repository called the function repository <NUM>. The network service provider can manage this function repository so the workflow manager or the NBMP (media) source <NUM> can lookup available media processing functions <NUM>, and then make a decision to run the functions <NUM> as media processing tasks on processing entities allocated for the workflow <NUM>.

In a network media processing system <NUM>, it is possible that one or more functions <NUM> need to be executed very frequently, i.e. the functions <NUM> are applied together in the same order in many of the NBMP workflows <NUM>. This can happen because: (<NUM>) one or more functions <NUM> need to be applied before or after another function <NUM> to make either the input <NUM> or output <NUM> respectively compatible with previous or next functions in the workflow <NUM>; (<NUM>) a split of a given processing functionality happens when a media processing functionality needs to be implemented using multiple media processing functions <NUM>; or (<NUM>) The processing requirements cannot be satisfied by a single function <NUM> and have to be distributed over multiple parallel instances of the function <NUM> of which the output has to be multiplexed. In any case, it is quite common that a set of functions <NUM> appear together for many of the media processing functionalities. To address this case, a function group <NUM> is defined and represented as such in the function repository <NUM>.

To define function groups in the function repository, the present disclosure provides a method in which the function repository holds a separate table or markup representing the grouping of different functions of one the group. In an embodiment of the present disclosure, the table or the markup is of the following format:.

Functions <NUM> can be grouped together in a function group <NUM> using the "seq" and "par" keywords as described above. When such function groups <NUM> are created in the function repository <NUM>, the responsibility of the function developer is to make sure that the output ports of a function <NUM> can be connected to the input ports of the next function <NUM> in the function group <NUM>, i.e. the functions <NUM> that are compatible with each other. However, if the two functions <NUM> that need to be grouped together are not compatible with each other (i.e., output ports of a function <NUM> cannot be connected to input ports of another function <NUM>), then the workflow manager <NUM> can insert one or more "compatibility functions" to connect those incompatible functions in a function group <NUM>.

With the above type of group information in a separate table/database, the grouping information is held separately from the function definition, and as a result the functions are not closely tied to a function group <NUM>, i.e. functions <NUM> defined in a function group <NUM> can also be used without other functions <NUM> in the function group <NUM> if required in a different workflow.

For the above table or database to be maintained in the function repository <NUM>, the entities or individuals who request registration of functions <NUM> inside an operator's function repository <NUM> should build appropriate function definitions using the representation of function definition using different descriptors described above. Also, the workflow manager <NUM> can construct this table based on function descriptors in the function repository <NUM>. This will be helpful when the functions <NUM> are registered by different vendors and had not prior intention of grouping their functions <NUM> with other functions <NUM> defined by a different vendor.

In certain embodiments, as an alternative to maintaining a separate function group table/database, each function <NUM> can express its group intentions using a "Group Descriptor". The Group Descriptor provides the grouping information of that function <NUM> with respect to other functions <NUM> in the function repository <NUM>. This Group Descriptor is included in addition to other descriptors as described earlier while providing a function definition for insertion into the function repository. The Group Descriptor provides grouping information using the "seq" and "par" as described earlier, along with a new keyword "self" as described below. For example, in function definition of func_2 (whose Id is func_d), the Group Descriptor can be shown using following examples.

Example <NUM>: func_2 in a sequence: Func_2 Group Descriptor: seq{<func_1_Id>,self, < func_3_Id>}, represents that func_2 (as represented using "self" keyword) is executed in sequence with func_1_Id and func_3_Id in order of sequence shown above.

Example <NUM>: func_2 in parallel: Func_2 Group Descriptor: par{<func_1_Id>,self}, represents that func_2 (as represented using "self" keyword) is executed in parallel with func_1_Id in the group.

Example <NUM>: func_2 in a parallel subgroup, but in sequence in a parent group: Func_2 Group Descriptor: seq{func_1_Id, par{<func_3_Id>,self}, func_4_Id} represents that func_2 (as represented using "self" keyword) is executed in parallel with func_1_Id in the sub group. And this sub group is executed in a sequence with functions func_1_Id and func_4_Id using the order shown by the grouping.

With these types of representation grouping of functions need to be performed during a function's definition registration in the function repository <NUM>.

In certain embodiments, a function <NUM> is able to indicate a list of function groups <NUM> that the function <NUM> may or must belong to. A flag is used to indicate whether grouping is essential or optional. The function <NUM> also shows its connections to other functions <NUM> of the function group <NUM>, number of instances of the function <NUM> to load in the function group <NUM>, the function's <NUM> position in the sub-workflow that represents the function group <NUM>. The information can be provided according to the following table.

In order to maintain synchronization of the media across the functions <NUM> of the same function group <NUM>, an additional metadata connection between the functions <NUM> is used. A synchronization signal is generated by the first function <NUM> in the function group <NUM> and sent as a separate metadata stream. The information is propagated to the last function <NUM> in the function group <NUM>, which can use this information to re-multiplex and re-sync the output of prior functions <NUM> in the function group <NUM>.

It is possible that when a function group <NUM> is created, one or more of the functions <NUM> within the function group <NUM> depend upon execution result of functions <NUM> that appeared earlier in the function group <NUM>. In this context, it becomes possible that functions <NUM> within the function group <NUM> not only exchange inputs and outputs, but also other data such as the configuration data and requirements data. One or more functions <NUM> in the function group <NUM> that appear early in the function group <NUM> can generate configuration data that the next functions <NUM> in the function group <NUM> might need for their execution. For this to be enabled, (<NUM>) functions <NUM> can generate and output configuration data that is sent as input to functions <NUM> that appear later in the function group <NUM>; (<NUM>) functions <NUM> can generate and output requirement data that is sent as input to functions <NUM> that appear later in the function group <NUM>; (<NUM>) functions <NUM> can generate and output monitoring data (e.g., quality monitoring data, security monitoring data etc.) that is sent as input to functions <NUM> that appear later in the function group <NUM>; and (<NUM>) functions <NUM> can generate and output assertion data (data that represents different checks that need to be performed) that is sent as input to functions <NUM> that appear later in the function group <NUM>.

Such data, for the benefit of functions <NUM> that appear later in the function group, can be generated and sent by the functions that appear earlier in the group. This data can be sent using the following two options. (<NUM>) Returning the data to the workflow manager <NUM>, which the workflow manager <NUM> can use to configure other functions <NUM> within the function group <NUM>. Typically static data can be sent to subsequent functions <NUM> within the function group <NUM> through the workflow manager <NUM>. (<NUM>) For dynamic data that is generated by the functions <NUM> earlier in the function group <NUM>, existing media and metadata channels can be used to send that information directly to the entities running subsequent functions within the function group <NUM>.

As described in the NBMP system <NUM>, a workflow description document is sent from the NBMP (media) source <NUM> for requesting set up of media processing in the network. The workflow description document may contain the list of functions that the NBMP source <NUM> intends to include in the NBMP workflow. With function groups in the function repository, the NBMP source <NUM> is able to insert function groups <NUM> in the workflow description document in addition to functions <NUM> in the function repository <NUM>.

The NBMP source <NUM> can insert function groups <NUM> in an NBMP workflow <NUM> using two different ways. (<NUM>) The NBMP source <NUM> can indicate a list of keywords based on which the workflow manager <NUM> can identify a function group <NUM> that is inserted in the workflow <NUM>. The workflow manager <NUM> can use the list of keywords to (a) can individually apply different keywords to search for functions <NUM> that match one or more of the keywords given by the NBMP source <NUM>. The workflow manager <NUM> can then group different functions <NUM> that match the multiple sets of keywords into a function group <NUM>; and (b) can decide on a function group <NUM> using relationships between different keywords given by the NBMP source <NUM>. (<NUM>) The NBMP source <NUM> can indicate usage of a specific function group <NUM> in the function repository <NUM> by including the specific function group <NUM> in the workflow description document that the NBMP source <NUM> sends to the workflow manager <NUM>. This option is preferred when the NBMP source <NUM> indicates the list of all media processing functions <NUM> that the NBMP <NUM> wants to be inserted in the workflow description document (i.e. source defined workflows). In this case, similar to a way where the NBMP source <NUM> indicates the list of functions <NUM> (e.g., using task connection map i.e. map of interconnecting tasks), the NBMP source <NUM> can include any function group <NUM> in place of a function <NUM> in the task connection map.

Different functions within a group may have clear dependencies with other functions in the group. The dependency information can be encoded using a set of keywords in the function repository as described below.

The function dependency information as shown above can be registered with the function repository while registering the function definition information. For example, this information can be registered using the Group Descriptor. The workflow manager will build group relationships based on this information.

As the function repository is maintained by the NBMP operator, it is entirely possible that the implementation of different functions in the function repository comes from different vendors i.e. different implementers. With this setup, it is entirely possible that a vendor might provide a function for a required functionality and a different vendor might provide a function group implementation for the same functionality. When these two types of implementations are available, it becomes the responsibility of the workflow manager to choose one implementation for a given request from the NBMP source.

When such multiple options are available in the function repository, it is proposed that the workflow manager treat this as a classification problem and use machine learning algorithms to choose the correct implementation to insert into the NBMP workflow. When the NBMP source clearly indicates the function or function group in the workflow description document, the workflow manager just includes the respective function or function group in the NBMP workflow. In this case, the workflow manager need not use the inference from classification model to find the correct implementation to insert in the workflow. However, when there is no indication from the NBMP source on the type of implementation to be included in the workflow (e.g., when NBMP source just sends keywords for choosing functions or function groups), then it is proposed that the workflow manager use the inferences it has derived using the learning algorithms that it runs to choose the correct implementation.

Choosing the correct implementation (e.g. whether to insert a function or function group, or which implementation among a varied set of implementations can be done as following: (<NUM>) Collect different implementations and treat each of them as a classification option i.e. an outcome. (<NUM>) Depending on a number of input variables, find which outcome (implementation) is more probable/feasible. This can be done using a number of classification models such as logistic regression, decision tree, random forest, gradient-boosted tree, multilayer perception, one-vs-rest, and Naive Bayes. Using any of the algorithms listed above, a number of input factors can be used for learning to find the correct outcome. The different input factors that the above algorithms can consider to choose the correct outcome (implementation) are: (a) NBMP source preferences such as requested function/function groups requested by the NBMP source. (b) Requirements passed by the NBMP source in the workflow description document. A number of input factors can be based on requirements information as listed below: (i) Bit rate requirements: Choosing an outcome based on requested bit rate. (ii) Throughput requirements: Choosing an outcome based on requested throughput. (iii) Hardware requirements: Choosing an outcome based on type of hardware requirements provided by the NBMP source such as the processor requirements, CPU cores, GPUs etc. (iv) Storage requirements: Choosing an outcome based on storage. (v) Security requirements: Choosing an outcome based on security requirements. (C) Operator preferences in choosing the correct implementation.

Based on a number of input factors as defined above, the workflow manager can classify a likely outcome that represents a given implementation (e.g., a function or a function group). The outcome represents the implementation of a functionality that the workflow manager inserts into the NBMP workflow.

<FIG> illustrates a workflow pipeline with multiple tasks, tasks <NUM>-<NUM>. Each task represents a function, similar to the function <NUM>, of <FIG>. The workflow manager <NUM> selects each of the tasks <NUM>-<NUM> in order to generate the output <NUM> based on a received workflow description from the media source <NUM>. Task <NUM>, <NUM>, and <NUM> can receive the input data, or a portion of the input data. In some embodiments, a function can receive multiple inputs such as the tasks <NUM> and <NUM>. In some embodiments, a function can generate multiple outputs, such as the task <NUM> and <NUM>. In some embodiments, functions can process data in parallel such as the tasks <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>.

Although <FIG> illustrate the environment-architecture, <FIG> illustrates an example workflow, <FIG> illustrates an example function, and <FIG> illustrates an example function group. Various changes can be made to <FIG>. For example, any number of functions can be included in the workflow <NUM>.

<FIG> illustrates an example method <NUM> for workflow generation with function groups in accordance with an embodiment of the invention.

In operation <NUM>, the workflow manager <NUM> can send a query to a function repository <NUM>. The query can include inputs, outputs, functionality, etc. of the function <NUM>. The function repository <NUM> can traverse a list of the stored functions <NUM> to determine one or more functions that include the requested inputs, outputs, or functionality.

In operation <NUM>, the workflow manager <NUM> can receive a list of matching function descriptions. The function descriptions can include the inputs, outputs, and functionality found through the traversal of the function repository <NUM>.

In operation <NUM>, the workflow manager <NUM> can determine if a suitable function <NUM> belongs to a function group <NUM>. The workflow manager <NUM> performs the determination for each function <NUM> in the list of matching function descriptions.

In operation <NUM>, the workflow manager <NUM> can receive one or more function group descriptions. When a function <NUM> is part of a function group, the workflow manager <NUM> requests the function group description from the function repository.

In operation <NUM>, the workflow manager <NUM> can build the workflow <NUM>. The functions <NUM> and function groups <NUM> can be added to the workflow <NUM> based on the function descriptions and the function group descriptions.

<FIG> illustrates an example method for management of network based media processing functions in accordance with an embodiment of this disclosure. For example, the process depicted in FIGURE <NUM> is described as implemented by the server <NUM> of <FIG>, the electronic device <NUM> of <FIG>, the media source <NUM> and the server <NUM> of <FIG>.

In operation <NUM>, the server <NUM> can receive a list of functions included in the functions repository. The list of functions can be received in response to a query from a media source. The list of functions can include function descriptions, function inputs, function outputs, etc..

In operation <NUM>, the server <NUM> can build a workflow using functions within the list of functions. The workflow <NUM> is generated based on requirements received from the media source in order to process a payload, such as a media file. The payload can be processed within a single entity or distributed amongst multiple entities. When building the workflow <NUM>, the resources and capabilities of the entities are considered to maximize efficiencies and minimize costs.

In operation <NUM>, the server <NUM> can determine an output of a first function and an input of a second function that are not compatible. When building a workflow, the server <NUM> can choose specific functions based on the function description. When optimizing the workflow, the server <NUM> can determine that certain functions, while optimal for the processing the payload, are not compatible with a prior function or following function. The determination of the compatibility can be based on an output of the first function and an input of the second function. The output of a function must be compatible with the output of the function immediately following.

In operation <NUM>, the server <NUM> can insert one or more compatibility functions between the first function and the second function. The compatibility functions can be a transform to handle the output of the first function and the input of the second function. The one or more compatibility functions can have multiple functions that are used. When multiple compatibility functions are used, they can be placed in the workflow in series or parallel depending on the functionality of the compatibility functions.

In operation <NUM>, the server <NUM> can create a function group with the first function, the one or more compatibility functions, and the second function. Once the compatibility functions are determined, the group of the first function, the one or more compatibility functions, and the second function can be grouped in a function group. The server <NUM> can provide a group description of the function group when saving in the function repository for future use or for replacing other instances of the first function and the second function in a workflow.

In operation <NUM>, the server <NUM> can replace each instance where the first function immediately precedes the second function with the function group. For existing workflows or other instances of the first function preceding the second function, the server <NUM> can determine that the function group would enhance the workflow by replacing. The server <NUM> could also determine that functions with similar descriptions or purposes to the first function and the second function could operate better using the new function group.

In operation <NUM>, the server can process a payload using the workflow including the function group. The server <NUM> directs the payload from the media source <NUM> based on the workflow <NUM> to process the payload for consumption by the media sink <NUM>.

Although the figures illustrate different examples of user equipment, various changes may be made to the figures. For example, the user equipment can include any number of each component in any suitable arrangement within the scope of the appended claims. Moreover, while figures illustrate operational environments in which various user equipment features disclosed in this patent document can be used, these features can be used in any other suitable system within the scope of the appended claims.

None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claim scope. The scope of patented subject matter is defined only by the claims.

Claim 1:
A method (<NUM>) performed by a workflow manager (<NUM>) in an electronic device (<NUM>), the method comprising:
sending (<NUM>), to a function repository (<NUM>), a query;
receiving (<NUM>), from the function repository (<NUM>), a function description for each of a set of media processing functions;
determining (<NUM>) that a function (<NUM>) in the received set of media processing function descriptions belongs to a function group (<NUM>), the function group being a set of connected media processing functions which are designed to work together in workflows, wherein output ports of a given media processing function in the function group are connected to input ports of the next media processing function in the function group;
receiving (<NUM>), from the function repository (<NUM>), a function group description including information on the function group;
determining a workflow based on the function group description; and
applying (<NUM>) the set of media processing functions connected in the function group for the workflow,
wherein the function group description further includes connectivity information on at least one function identifier of the set of functions for a connection.