Patent Publication Number: US-2020279304-A1

Title: Content management system and method for operating same

Description:
FIELD 
     The present disclosure relates to Content Management Systems, in particular, though not exclusively, to methods and systems in which a catalogue of digital content is provided to a user in order that they might select the digital content for consumption, for example via use of a graphical user interface. 
     BACKGROUND 
     Many systems exist which allow users to browse and select digital content for consumption. Users are often presented with a digital catalogue representing a set of digital content items which they can view on a computing device, or on a television, via a set top box. The catalogue will usually be logically ordered in such a way as to allow the user to intuitively navigate to the content which they are looking for. For example, in the context of a catalogue of videos, the videos may be categorised into type/genre, with more popular videos being presented to the user in prime position, say at the head of a list, or simply more prominently than other items. In a system having a well-ordered catalogue, the user should be able to navigate to the items they wish to find easily and efficiently. There is a need to provide an improved content management system. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a system comprising a content management system in communication with a plurality of user devices via a communications network; 
         FIG. 2  illustrates a Content management server in further detail; 
         FIG. 3  illustrates a content management terminal in further detail; 
         FIG. 4  illustrates a user device in more detail; 
         FIG. 5  illustrates a process for identifying content items in a content database which suitable for promotion; 
         FIG. 6  illustrates a further system comprising a content management system in communication with a plurality of user devices via a communications network 
         FIG. 7  illustrates a process for identifying content for promotion in the context of a content management system; and 
         FIG. 8  illustrates a process of correcting irregularities that are discovered prior to identifying the content to be promoted. 
     
    
    
     DETAILED DESCRIPTION OF THE DRAWINGS 
     In overview, disclosed methods of operating a content management comprise collecting or receiving data which relates to a user&#39;s behaviour when using a digital content catalogue, and prior to the consumption of a content item. This collected or received data is processed and analysed to create a data table which may be used to demonstrate to a content manager which items in the digital content catalogue are most difficult to discover in order that the catalogue be re-ordered appropriately. In this way, user behaviour may be used to optimise digital content management systems. Once it is known, for a given digital content item, how discoverable, or indeed undiscoverable it is based on a user&#39;s behaviour prior to consuming the content item, that item may be promoted to a more prominent location in the catalogue or relocated to a different location so as to allow a user to more easily locate it. 
     In some aspects of the disclosure, a method is provided, wherein a plurality of digital content consumption notices, comprising discovery data for one or more content items in a first digital content catalogue, are received at a content management system. The discovery data relates to a user usage type or user behaviour. A first data structure is created, which includes digital content metrics, which themselves include the discovery data for one or more of the digital content items. Subsequently, the digital content metrics are extracted to thereby create a second data structure. Creating the second data structure includes a ranking of one or more digital content items on the basis of the digital content metrics. Instructions are communicated to a user device to cause display of the second data structure. 
     Advantageously, through the analysis of the received discovery data through the creation of digital content metrics a ranking of the content items is established. This ranking is based on discovery data which can be used to create a discovery value for a content item over time. The discovery values for the content items can be given a ranking which, when provided to a content manager, allows the content manager to update a content catalogue by promoting content which has been determined to have a low discoverability value. 
     In some embodiments, the digital content metrics for a given content item or items comprises an association, which may be an index or indicator, or link to one or more user categories. 
     In some embodiments, the one or more user categories are predefined. For example, a user category may be a user cluster, where the user cluster may be based on a target demographic or sector of users, where the target demographic includes demographic features that may be based on information in a user profile that a user has provided to a content provider. In some embodiments, a user category (which may be a user cluster) may be alternatively or additionally based on demographic features learned and/or gathered (such as using a known learning engine such as an artificial neural network) from tracking or monitoring a user&#39;s behaviour or usage type associated with a plurality of content items from the media content service provider. In related embodiments, one or more usage patterns based on similar user behaviour or usage types is associated with a number of user categories, where the user categories are user clusters share at least one common demographic feature. 
     Advantageously, associating the digital content metrics of one or more content items with one or more user categories, each relating to a user cluster for instance, provides an indication of the target user sector or demographic information intended for the one or more content items. The target sector or demographic may be previously associated with each content item based on existing or reference data (demo data) or may be associated dynamically based on regular usage patterns identified for the one or more content items. 
     In some embodiments, the digital content metrics of a content item includes or may be associated with metadata of the content item. The metadata of the content item may, for example, include one or more metadata parameters. One such parameter may relate to a content category of the content items. For example, if the content item is a children&#39;s movie, then the metadata parameter for its content category may “kids” or “family”. 
     In some embodiments, the association to the one or more user categories for a content item or items in the digital content metrics is based on at least one metadata parameter relating to the given content item or items. In related embodiments, the digital content metrics for a given content item or items comprises an association or link at least one other digital content item having the same metadata parameter. In some embodiments, the discovery data for the given content item is related to at least one such association. 
     Advantageously, associating a user category, for example a user cluster, with a metadata parameter for a content item ensures that the content item is automatically provided in a digital content catalogue for users belonging to the user category or cluster that matches at least one target demographic feature. Hereinafter the term user cluster is understood to refer to a user category and is used interchangeably throughout the disclosure. 
     In some embodiments, the user usage behaviour or usage type may be broadly broken down into “natural” or “search” behaviour. The behaviour which falls into the “natural” category includes normal browsing behaviour using a user interface, and “search” behaviour is so-called since it relates to the user entering search terms into a search interface. A content management server is arranged to order the received data so as to rate the items in a digital content database according to how easy or difficulty with which users have discovered them, prior to their consumption. The discovery data may be collected during a first period having a start time and an end time, the period starting when a user initiates use of the user interface when it has been inactive for a predetermined time. The time period may end when the user ultimately consumes a piece of content. 
     Digital content items may be identified for promotion based on the ranking. Once the items have been identified for promotion, a second digital content catalogue may be created, based on the identified digital content items. The digital content catalogue may be communicated to a user device. The extracted digital content metrics may be used to calculate a discovery ratio of user usage type for the content item to which the metrics relate. Advantageously the discovery ratio allows the content manager to understand for each item of content how well placed it is for intuitive of natural discovery rather than the need to enter partial or full search terms in order to find specific content. 
     In some embodiments, the discovery data received may include further collected user usage data. For example, it may include the number of user inputs into user interface prior to consumption of a content item, the number of searches prior to consumption of a content item, the duration of browse before initiating a search, or an indication of an unsuccessful search followed by the initiation of browse. The metrics can therefore be further enhanced by monitoring not only the type of user interaction with the system, but also the features of the interaction, such as duration, etc. outlined above. 
     A further discoverability factor may be calculated which includes the total number of discoveries for a content item. Advantageously including the number of discoveries for each content item allows for a discovery value to be calculated and applied to the ranking. The discovery value indicates whether a content item is popular or not, and whether it is being discovered naturally. 
     In some embodiments, based on a determination that a content item is not being discovered naturally but is still being consumed by users of one or more user clusters, then a discovery alert is generated. In some embodiments, the generation of the discovery alert is triggered by one or more events, such as deviations or irregularities from expected user behaviour based on predefined, known or learned historic data related to a given content item and/or user category. Therefore, if it is determined that the content is not being discovered naturally, i.e. discovered based on a user conducting a search for the content, then the presence of one or more of the above events is monitored before a discovery alert is generated. In other embodiments, the discovery alert is generated automatically soon after the user searches for the content. 
     In some embodiments, a determination may be made if a usage type by a user associated with a particular user category satisfies a criterion, which may be predefined. For instance, the user type may be a search or a navigation(browsing) action for the content item, as mentioned above. In some cases, the user using the digital content catalogue may belong to an older demographic or the user may be using or associated with a guest or temporary profile or account. In such cases, a defined criterion may need to be satisfied, before discovery alert is generated. 
     For example, if the usage type is a search action and the user category is related to a demographic associated with a main user account or profile, then the defined criterion is considered to be satisfied and a discovery alert is issued to indicate that a given content item is not being discovery by the user naturally, as the main user is having to search for it. In another example, even if the usage type is natural, i.e. a navigation action, for the same main user category as mentioned in the example above, but the discovery data indicates that such navigation or browsing action is carried out for more than a certain amount of time, such as 30 seconds or 1 minute, then the defined criterion is satisfied. In these cases, a discovery alert is issued to indicate that the user is searching for certain content that is not appearing at a prominent position in the digital content catalogue. 
     On the other hand, in some embodiments, even if a content item is being searched for instead of being discovered naturally (so the usage type is a search action), a discovery alert is not generated if the user category is a temporary/guest user or is one associated with users belonging to an older age group demographic. In this case, the defined criterion is not considered satisfied and thus there will be no discovery alert generated. In another example, if the user category is one associated with the main user but a one-off search is carried out for an unusual or temporary content category, i.e. something that the main user would not usually search for, then the defined criterion is not satisfied, and no discovery alert is generated. For example, if the user searches for content relating to a holiday destination (as he or she may be planning to visit such destination) for a first time, this search action will not result in an alert being generated to ensure that such holiday destination always appears in the user&#39;s digital content catalogue. 
     Advantageously, the above embodiments first check if a criterion related to a defined or an appropriate user category and/or a defined or appropriate content category is satisfied, irrespective of the usage type identified. If the criterion satisfied, then a discovery alert is generated thereby ensuring that a content manager or content management system is efficiently generating and/or processing such discovery alerts when it is considered appropriate or necessary. Thus, this advantageously avoids discovery alerts being generated every time a particular usage type, such as a search action is carried out by any given user. 
     When generated, the discovery alert may include a flag or setting or reference(index), where the discovery alert is related to metadata of the content item or the association with the one or more user categories or clusters. In some embodiments, besides being associated with the content item, the metadata may also be present in or obtained from the digital content metrics. 
     In some embodiments, the discovery alert is an irregularity notice, for notification of irregularities or unusual patterns as mentioned above. 
     Advantageously, the above described embodiments enable unusual or unexpected user interactions with content items to be tracked, so that further investigation can be carried out as to the reason for such irregular interactions. 
     For example, considering that the discovery alert is an irregularity notice, such irregularity notice may be generated if there is a low interaction rate for an expected target user cluster for a content item, or a high interaction rate for an unexpected user cluster, or a low interaction rate as an absolute for a content item that was previously popular or linked to a content item that is usually popular. In related examples, whether or not a given cluster is expected or unexpected for the content item may be based on a prior determination of historic user behaviour or interaction, such as historic viewing data or reference data (demo data). In other related examples, if the content item has a low absolute then this may be notified in the irregularity notice as an unusual content item that may need further verification. 
     Further examples of irregularities, deviations or events for triggering generation of an irregularity notice include one or more of:
         An unusual start time or stop time associated with the content item as an absolute, for instance if a children&#39;s program is viewed after 10 pm often. This can suggest that the content item may be incorrectly advertised or described as if it is a different type of program or content and is thus not being viewed by children during the day.   An unusual start time or stop time for the content item associated with users in a user cluster for which the content item or a linked content item was previously popular, for instance if a second season of a television series that was popular for a particular user cluster is now being exited early or even straight after the introduction. This can suggest that the content item may be incorrectly advertised or described as if it is a different type of program or content and may not actually be the second season (it may instead be the first or the third season, or a completely different television series). In a related example, the content item may be an advertisement that is targeted for a particular user cluster. Therefore, if it is determined that the advertisement is exited early or stopped, when previous responses to the advertisement or similar advertisements were positively received by the same cluster, this can suggest that the advertisement has not reached the intended cluster.   The content item has a low interaction rate on a particular type or types of rendering or playback device for a user cluster but not on other devices for the same cluster, for instance if unusually a movie is not watched at all on mobile devices such as smartphones and tablets or PDAs and is only watched on televisions by the same user cluster. This can suggest that the content is not displaying on mobile device due an error in the playback format or other technical error associated with the content item itself, or that there is some indication that it may be unavailable on the mobile devices.       

     Related to the above embodiments, a step of updating the digital content metrics and/or the one or more user categories or clusters is performed, based on the discovery alert. In some embodiments, the step of updating the digital content metrics includes correcting the metadata parameter and/or its association with the content. In other embodiments, the step of updating includes correcting the user clusters and/or one of more demographic features on which the user cluster is based. 
     In some embodiments, upon generation of the discovery alert such as the irregularity notice, it is verified if the metadata or a metadata parameter of the content item is associated with one or more user clusters that matches the association or link in the digital content metrics for the content item. 
     Advantageously, this enables a verification of whether the content item is being provided or promoted to the correct or intended target audience represented by the user clusters. 
     In some embodiments, upon generation of the irregularity notice, additionally or alternatively it is verified if the metadata or metadata parameter is applied to the correct content item. 
     Advantageously, this enables a verification of whether the metadata is linked or “tagged” to the correct content item. For instance, it is verified if the description of the content item is linked correctly to the respective content item, or if it linked to another content item in error. 
     In some embodiments, upon generation of the irregularity notice, additionally or alternatively it is verified if the one or more clusters are associated with correct demographic features. 
     Advantageously, this enables a verification of whether the metadata or user clusters have been assigned consistent and/or correct demographic features, or if the demographic features have been applied to another user cluster in error instead of an intended user cluster. 
     Related to the above embodiments, if it is determined that the verification is unsuccessful, then a step of updating the metadata parameter and/or the association in the digital content metrics and/or the one or more clusters is performed, based on a verification outcome. In some embodiments, the step of updating includes correcting the metadata parameter and/or its association with the content. In other embodiments, the step of updating includes correcting the user clusters and/or one of more demographic features on which the user cluster is based. 
     In some embodiments, the above described ranking of the content items is established after the verification takes place, said verification conducted when an irregularity notice is generated. Furthermore, the ranking is established after the step of updating is conducted for the user cluster or the metadata parameter. Then, following this ranking (after the steps of verification and updating, if applicable), the content items are identified for promotion based on the ranking for creating the second digital content catalogue for the user device, as mentioned above. 
     Based on the above description of the embodiments, a further aspect of the present disclosure relates to a method comprising receiving, at a content management system, a plurality of digital content consumption notices comprising discovery data for one or more content items in a first digital content catalogue. The discovery data relates to a usage type of a user interface, the usage type being one of a navigation action comprising navigation of the first digital content catalogue via a user interface to consume a content item in the one or more content items, or search action comprising entering of search terms into a search interface to consume the content item. The method includes creating a first data structure including digital content metrics, the digital content metrics including the discovery data for the one or more of the content items. The digital content metrics for each content item of the one or more content items also comprises an association to one or more user categories. Then, based on a determination of a search action for a given content item among the one or more content items, a discovery alert is generated. In some embodiments, the discovery alert may be an irregularity notice. In some embodiments, the discovery alert relates to metadata of the given content item and/or the association to one or more categories for the given content item. Then digital content metrics for the given content item is updated based on the discovery alert. 
     In some embodiments, responsive to the generation of the discovery alert, the digital content metrics is updated for the content item based on the discovery alert. In some embodiments, the metadata and/or association in the discovery alert is verified. In related embodiments, upon a determination that the verification is unsuccessful, digital content metrics for the given content item is updated based on an outcome of the verification. 
     The disclosed method includes the steps of extracting the digital content metrics of the one of more content items, including the updated digital content metrics for the given content item. A second data structure is then created based on the extracted digital content metrics, wherein creating the second data structure includes a ranking of one or more digital content items on the basis of the extracted digital content metrics. The method then includes communicating instructions to cause display of the second data structure at a content management terminal. 
     Optionally, the method further comprises periodically receiving a list of key terms relating to aggregated user data. Upon receipt of a list, a search of the content items is performed, and a weighting is added to digital content item information in the second data structure, the weighting based on the correlation between the key terms and digital content item metadata retrieved in the search. The ranking of the digital content items may be based on the digital content metrics and the weighting. By applying the extra steps of including key terms, such as trending online data, the ranking of content items can advantageously be made even more relevant and pertinent to a user, based on a combination of the behaviour of users of the system and users of external systems 
     Further aspects of the disclosure relate to a system comprising a content management server arranged to implement the disclosed methods. The content management server may receive digital content consumption notices from a user device which is arranged to monitor user behaviour in order to collect discovery data for consumed content items. 
     A digital content item, is any video, music or textual content which is viewable by a user. The terms “digital content item” and “content item” are used interchangeably throughout this disclosure. 
     Discovery data is data relating to the behaviour of a user in the period prior to consuming an item of digital content. Various examples are provided herein. Usage type or user behaviour is one element of discovery data. User behaviour is defined herein as relating to either a “natural” discovery or a “search” discovery. A natural discovery is defined as one in which the user has utilised the standard functions of a user interface to locate the content item such as moving a selector using up, down, left, and right controls, for example. These standard functions may involve the use of other input devices, for example a computer mouse, or TV remote control. A search discovery is one that has involved the user entering full or partial search terms into a search field presented to the user. A digital content item is an item of content which is presented to a user via a content catalogue. Examples of digital content include, but are not limited to: media content such as video or music, textual content such as books, or newspaper/magazine articles, or similar. An item of digital content is deemed “consumed” when a user takes the necessary action to view that content. For example this may be selecting a visual icon representing the content which is present in a visual representation of the catalogue presented to a user, or touching a playback button provided on touchscreen display so as to commence playback of a video. “Consumption” of a content may be defined as “engaging” with the content; it is not required that a video be played until its conclusion, or a text item be read to the end. By way of example a content item may also be deemed “consumed” at the point in which it is added to a playlist or wishlist, for later consumption by the user. A data consumption notice is therefore an indication that the user has engaged with a content item, and the discovery data provided in the notice gives information about the manner in which a user arrived at their engagement with the content item. 
     Some specific embodiments are now described by way of illustration with reference to the accompanying drawings in which like reference numerals refer to like features. 
     With reference to  FIG. 1 , a system  100  comprises a content management server  102 . The content management server  102  is in communication with a content management terminal  104 . Both the content management terminal  104  and content management server  102  are in communication with a data collector  106 . Content management terminal is in communication with client user devices  108  via communications network  110 . In some embodiments, the content management server  102  and the content management terminal  104  are combined. 
     The data collector  106  receives data from client user devices  108  relating to user usage behaviour via communications network  110 . A client user device  108  may be a set top box connected to a television or other display device, or may indeed be any device on which it is possible to view digital content. Such devices may include computing devices such as personal computers, laptops, tablet devices, and mobile devices such as smartphones or PDAs, or e-readers. 
     The client user device collects data relating to user behaviour prior to consumption of a digital content item. This data is forwarded to data collector  106  periodically, or at a time related to the consumption by the user of a content item at user device  108 . The data collector provides content management server  102  with discovery data which includes various determined user behaviour metrics, described more fully below. The content management server  102  receives the discovery data and associates the data to content metadata located in a content metadata repository. The discovery data, including usage behaviour metrics, can be used to update content metadata in the content metadata repository. The content metadata repository stores information relating to content items in a content library. Digital content metrics (as explained above) are created from the discovery data, the digital content metrics relating the user behaviour, this further relates to the behaviour of users in one or more user clusters, to the specific item in the content library which is deemed consumed by the user. Periodically, a calculation of the “discoverability” of the content items is carried out based on the metrics, and each content item for which data is available is assigned a discoverability ratio and/or a discoverability factor. Once the calculation is complete, and in some embodiments, after the step of verification following generation of a discovery alert such as an irregularity notice, and if applicable, a step of updating the digital content metrics based on the irregularity notice is completed. Then a ranking of the content items is created and may be sent to content management terminal  104  for display to a content manager. The content manager may then decide to promote a given content item based on its position in the ranking and the discoverability factor. Hereinafter the terms discovery alert and irregularity notice are used interchangeably to refer to such alert following a detection of a search action for content. 
     With reference to  FIG. 2 , some embodiments of the content management server  102  are now described. The content management server  102  comprises a processor  202 , in communication with a memory  204 , a content metadata repository  206 , a network interface module  208 , a usage data calculation module  210  The processor  202  is arranged to coordinate between the content metadata repository  206 , the network interface module  208 , usage data calculation module  210 . The memory  204  can store instructions for execution by the processor  202  to cause the processor  202  to provide desired functionality. 
     The network interface module  208  is arranged to receive discovery data output from the data collector  106  and pass it to usage data calculation module  210 . The network interface module is further arranged to send data from the content management server  102  to the content management terminal  108 . Usage data calculation module  210  receives the discovery data including usage behaviour metrics and associates the data with metadata stored in the content metadata repository  206  in order to create a data structure including discoverability data for each content item. The data structure may take the form of an ordered list which highlights the digital content items which have a low discoverability factor. 
     Usage data calculation module  210  ingests the discovery data including usage behaviour metrics, and over a given timeframe or time period, maintains a tally of the number of consumption notices relating to consumption of a content item for which metadata is stored in the content metadata repository. The irregularity notices in some embodiments are also considered for this tally. The tally relates the number of consumption notices which are determined to relate to natural discoveries or search discoveries. The usage data calculation module periodically calculates a ratio of the number of natural discoveries to the number of search discoveries to provide a “natural discovery ratio”. This is defined as: 
       Natural discoveries/search discoveries=natural discovery ratio. 
     Lower ratios for given content items indicate that the content items are difficult to discovery naturally, without the use of a search function. In this way, the content management server  102 , together with the data collector calculates a discovery ratio for all of the content items in the system. 
     One enhancement to the discoverability ratio factors in the total number of views into the calculation. This not only provides the ability to identify what the discoverability was but also the volume of discoverability. This can be calculated as follows: 
       (Natural discovery/search discoveries)*total number of discoveries=discovery value, 
     where large numbers indicate that the content is popular and being discovered naturally. Small numbers indicate that the content is unpopular or not naturally discovered. The usage data calculation module may also compare the total number of discoveries (or consumptions) within a certain period and compare this number to a baseline number for a corresponding period which is stored in the content metadata repository. If a content item is consumed significantly more than the baseline this would indicate that a content item has become recently more popular. A content item that is determined to satisfy an upsurge in popularity also has a low discovery ratio or discovery value, this indicates that the content item is not located in a place in the digital content catalogue which is conducive to being discovered naturally. This may be determined optionally or additionally based on generation of irregularity notices, prompting further verification and updating, if applicable. When such a determination is made, the content item may have a data flag applied to its entry in the list indicating that it is a candidate for promotion. 
     The content metadata repository  206  contains metadata regarding the content items which are present in the system. As stated previously, digital content may include video content, including feature films and television series. The metadata stored for the content items includes links or associations, as mentioned previously between a content item and other similar content items. For example, the episodes of a television series will be associated with one another, and films forming part of a series may be linked or grouped. When creating the data structure, the usage data calculation module  210  may evaluate discrepancies in discovery values and ratios for linked or grouped content items. When creating the list, the discovery ratio or value for a content item may be compared with the ratio or value for a linked content item. Where a difference exists which is larger than a predetermined size, a data flag may be applied to the content item or linked item having the lower of the two ratios or values, indicating that the content item is a candidate for promotion. 
     As mentioned above, the usage data calculation module  210  creates a data structure comprising an ordered list of the content in the system. This list may be created periodically, for example daily, or even hourly. The list may be created on receipt of a user request. The list may include metadata relating to a popularity surge or a discrepancy between content items linked as part of a group, as outlined above. Such metadata may be used to flag to a content manager that a content item is particularly suitable for promotion. The list may also include updated metadata parameters following the verification that takes place responsive to the generation on irregularity notices. 
     Once created, the ranked list is communicated to the content management terminal  104 . 
     With reference to  FIG. 3 , some embodiments of the content management terminal  104  are now described. The content management terminal  104  comprises a processor  302 , in communication with a memory  304 , a content catalogue module  306 , a network interface module  308 , a display  310  and an input device  312 . The processor  302  is arranged to coordinate between the content metadata repository  306 , the network interface module  308 , display  210  and input device  312 . The memory  304  can store instructions for execution by the processor  302  to cause the processor  302  to provide desired functionality. 
     The content management terminal  104  receives a data structure from content management server  102 . The data structure comprises the ranked list of content including metadata determined from the discovery data metrics by the usage data calculation module  210 . The data structure may be communicated with instructions causing it to be displayed to a content manager via display  310 . A content manager may review the transmission from content management server  102  and determine which of the flagged content items or content items which have low discoverability values or ratios are to be promoted. The content manager then updates or otherwise modifies a content catalogue stored in content catalogue module  306 . Updating or modifying the content catalogue may involve re-ordering content items within the catalogue, or applying new metadata to a content item in order that it be displayed differently upon rendering of the catalogue. The content catalogue may store one or more versions of the content catalogue, or may comprise instructions which cause the rendering of the catalogue at a user device upon user initiation of a user interface. The catalogue may then be constructed or otherwise built in dependence on the instructions, and further user metadata related to the user&#39;s usage habits or other information known about the user. In this way, a global catalogue may be provided to all users of the system, or tailored catalogues may be provided to specific users, for example based on user location, age, or other demographic information or demographic features. 
     If it is determined by the usage data calculation module  210 , for example, that the second film in a film series has a low discovery ratio and a low discovery value, as defined above, whereas the first film in the same series has received a surge in popularity, and has a standard discovery ratio, the content manager may modify the content catalogue so as to promote the second film to a location in which it is more naturally discoverable, so that users are more likely to naturally discover and consume a content item which is related to the already popular first film. The content manager may also promote the first film based on the popularity surge. Furthermore, upon triggering of an irregularity notice in the above situation, the content catalogue is in some embodiments, promoted after verifying the reasons for the irregularity. This verification is carried out to check for or rule out if there are any errors in the target user clusters, demographic features associated with the user cluster and/or metadata parameters relating to the metadata of the content items. 
     The content management terminal  104  may form part of the content management server  102  itself. The identification of items for promotion may be carried out automatically based on the values of discovery ratio and discovery value which are determined or based on verification outcomes after generation of an irregularity notice. In this case the content catalogue module  306  may automatically create an updated version of the digital content catalogue with the items identified for promotion moved into more prominent or discoverable locations, in order that the user has a better chance of naturally discovering them. The updated catalogues may be created periodically, or be created where a comparison of discovery values exceeds a threshold, which indicates that an item is particularly popular and is in need of promotion. Once a new catalogue is created by the content catalogue module  306 , it may be pushed out to one or more user devices  108  via the network  110 . 
     With reference to  FIG. 4 , some embodiments of the user device  108  are now described. The user device  108  comprises a processor  402 , in communication with a memory  404 , a control device  406 , a network interface module  408 , a display  410 , an input device  412  and a user behaviour monitoring module  414 . The processor  402  is arranged to coordinate between the content metadata repository  406 , the network interface module  408 , display  410  and input device  412 . The memory  404  can store instructions for execution by the processor  402  to cause the processor  402  to provide desired functionality. 
     User device  108  may be a set top box, or other electronic or computing device on or through which a user may view digital content. The user of the device is presented with a digital content catalogue, viewable on display  410 . The user interacts with the digital content catalogue through means of a user interface. The user interface may be navigated through use of the control device  414 . A user interface may present a user with the option to move a cursor up, down, left, right, or back through a presented array or list of content items, prior to selecting a content item, as is well known in the art. The control device  414  may be a mouse, connected wirelessly or via a wired connection to the user device  108 , or be a remote control device not forming an integral part of user device  108 . In the event that a user is unable to locate a desired content item through the use of the natural browsing environment of the user interface of the digital content catalogue, the user may initiate a search function. The search terms are input through the input device  412 . A search function most generally includes the entry of all or part of a text string. Input device  412  may be a keyboard, or may be specific buttons provided on the remote control device described above. Browsing control and input functions may also be provided via a touchscreen interface (not shown) of an example user device. 
     Data on a user&#39;s usage of the digital content catalogue is monitored and collected by user behaviour monitoring module  414 . The collected data is sent to data collector  106  through network interface module  408 . The sending of user usage data may be done periodically, based on a measured time interval since a session start time being exceeded, as discussed below. User behaviour may be logged as individual commands entered into the control device in the natural browsing environment. A “left” command may be logged as a single event. The sending of user usage data may be done when a particular number of events is reached. The memory  404  may comprise a local cache in which a limited number of events may be stored, prior to being communicated to data collector  106  through network communication interface  408 . 
     An important part of the invention is the starting point of the monitoring and collection of data during the content discovery process, and the content name/ID of the content consumed as a result of the discovery process. For example, when the user device  108  has been at rest for a predetermined period, which may be a number of minutes, and there has been no user interaction, the user interface is considered to be ready for a fresh content discovery session. When the system is considered ready for a fresh content discovery session, the first user interaction with the user device will cause a session start event to be logged. The discovery data event end is determined as when the user consumes a media content item, and the discovery data may therefore comprise both natural browsing and search event data. 
     In an example scenario, a first user begins a session at the user device  108 . This takes place after a predetermined system idle time has been reached, and so a new session event is logged. The user begins naturally browsing through the content catalogue, using buttons on their control device to directionally navigate through the system. Each time the user enters a browsing instruction, this is logged as an event. After a period of natural browsing the user selects a content item for consumption. This may be by pressing a “play” button to start a video in the caser that the content catalogue is a video catalogue. This is logged as a session end event, and the discovery data is communicated to the data collector through the network communication interface. The data collected by the user device is analysed by the data collector and defined as a natural discovery event, which is associated with a content name or content ID included in the session end event data. 
     A further user using a different user device  108  similarly begins a session, also taking place after a predetermined system idle time has been reached, which is logged locally at the user device as a new session event. The further user commences naturally browsing, and a number of browsing events are logged, however the further user turns to a search function, selecting a search function, for example displayed on the display, and enters a partial name for a content item. The user then selects an item returned by the search for consumption. At this point, the session end event is created, and the discovery data is communicated through the network communication interface to the data collector. The data received by the data collector from the user device  108  is analysed by the data collector and defined as a search discovery event, because of the use of search. The search discovery event is also associated with the content ID or name of the consumed content, present in the session end event data. 
     The data collector continues to collect a plurality of such discovery events from the user devices present in the system during a predetermined time period. Each of the events may comprise metadata which includes the amount of browsing data, nature of the browsing, time data indicating length of browsing before a user resorted to a search. The data collector forwards the data collected over the predetermined time period to the content management system  102 , which then computes, for the content items, the discovery ratios and values for the content items. 
     Referring to  FIG. 5 , a method of operating a content management system is described. At step  502 , consumption data is received. The consumption data relates to the consumption of a content item at a user device, following user selection of the content item. The consumption data comprises discovery data relating to the manner in which the user arrived at the selection of the content item. This discovery data may be categorised as natural browsing event data or search event data. The discovery data relates to a discovery session which is deemed to have started after a predetermined time period has passed since previous use of a user interface at the user device. The consumption data is received at data collector  106 . Alternatively, the data collector may form part of a content management server  102 , which receives the consumption data. Consumption data may be received from one or more user devices, and relates to a plurality of consumption events. 
     At step  504 , the data collector creates a first data structure including digital content metrics. This data structure is created from the discovery data events which are received from the user device. A separate entry is created in the data structure is created for each item of digital content which has been consumed, whether irregularity notices are generated and if so, the outcome of the subsequent verification, and digital content metrics are created as part of the data structure. These digital content metrics may include but are not limited to the number of browsing events, the browsing event types, duration of browse etc. 
     At step  506 , a second data structure is created from the digital content metrics extracted from the first data structure. The second data structure comprises a list of all of the content items which have been consumed and comprises calculated discoverability values and or factors for each content item in the list. 
     At step  508 , the second data structure is communicated to a content manager for display. This presents the content manager with an intuitive basis on which to select content for promotion. The second data structure may also comprise flagged content which is deemed particularly suitable for promotion. This may be based on any updates made following verification to correct a user cluster, metadata parameter or demographic feature 
     In an embodiment, content management system may receive information regarding subjects or keywords which are subject of a temporary spike in consumer interests. The information is received from one or more external sources in the form of a hotword list. The receipt of consumer interest or trend data allows for content managers to identify content which could be subject to promotion to an end user of the content management system. Such a temporal spike in consumer interest may be caused by news, events, or anything which causes names, content or subjects to trend. 
     Referring to  FIG. 6 , some embodiments of a content management system and environment will be described. System  600  comprises a content management server  602 , content management terminal  604 , hotword list generator  606 , and a plurality of end user devices  608  connected which communicate with the content management system and content management terminal via means of communications network  610 . The features of the components forming the system  600  are analogous to those shown in  FIG. 1  and described above with respect to  FIGS. 2 to 4 . 
     Content management server  602  receives one or more hotword lists from hotword list generator  606  via means of communications network  610 . Hotword list generator  606  can create hotword lists using a variety of techniques, including monitoring trending topics on social media platforms such as Twitter®, and website scraping. The hotword lists received from hotword generator will comprise trend weighting information to indicate how “hot” a particular term is, that is to say just how on trend a particular term is. 
     The content management server  602  uses the hotword list and matches content in a content database with the terms in the list. Each item in the hotword list is used as the basis of a search for matching content. The system iterates through each term in the hotword list, searching the database for matching content, and provides a relative weighting of the quality of the match between the term and the list of content items which are returned as a result of the search. 
     Therefore, for each hotword in the hotword list, a secondary list of matching content is provided. By using a combination of the trend weighting and the match quality weighting the system creates a data structure including a recommended list of content items for each hotword, identified as suitable for promotion. One or more lists may be provided, relating to one or more hotword terms. The list of recommended items may be provided to the content manager periodically, for example daily, or where a particularly strong weighting of trend and match is combined, an alert may be sent to the content manager. A content manager is able to review the list or lists in order to determine whether to promote one or more items from the list to the end consumer. The promotion of content items may take place through the creation of an updated catalogue of content which is communicated to the users. The content catalogue may be communicated globally to all end users of the system, or on the basis of user information known about each user. 
     The creation of a new content catalogue including promoted items identified by the creation of the recommend list may be carried out automatically. This new content catalogue may be presented to a content manager, or may be automatically communicated to the end users as described above. 
     The hotword list is ingested into the content management system through a rest-like API. 
     As shown in  FIG. 6 , this embodiment may be used to provide updated content catalogue to end users based on trend information. It may also be combined with the system shown in  FIGS. 1 to 4 , and described above, to provide an improved content management system. 
     In this case the list of content items recommended from the search of content matching the hotword lists is cross referenced against the content items which are identified as suitable for promotion based on the user interaction with the content catalogue via the user interface at the user device. Since both the hotword lists and discovery data both relate to user interest, this system allows for information gathered from both within the content management system and external to the content management system to be combined in order to provide the user with an improved system. 
     Referring to  FIG. 7 , a method of operating a content management system is described. 
     In step  702 , a list of hotwords is received at a content management server  602  from an external source. The hotword lists include weightings for each term indicating the strength of trend data associated with the term. It may be, for example that the hotword list comprises a list of actors mentions from a social media platform within a 24 hour period. The number of mentions will be reflected in the weighting applied to the hotword. 
     In step  704 , a search of a content database is performed for each item in the hotword list. The search may be performed by a SOLR search engine. 
     In step  706 , a list of matches from the search results is generated for each of the items by the SOLR search engine. The list of matches is ordered by relevance to the hotword. In the case that the hotword is the name of an actor, for example, video content featuring that actor will be highly relevant. 
     In step  708 , a list of content which is deemed relevant is presented to a content manager. The content manager then may selectively promote content in a digital content catalogue to be communicated to a user, in step  710 . 
     The hotword lists may be regularly ingested into the content management server  602 , and the subsequent steps of the method performed each time a list is ingested. This provides a constantly changing and updated list of recommended content for a content manager. 
     The hotword lists can be used over time to iteratively improve metadata associated with content items. The search results may be used to improve the relevance of terms associated with digital content in a content metadata repository. 
       FIG. 8  illustrates a process of correcting irregularities that are discovered prior to identifying content to be promoted in  FIG. 7 . 
     In step  802 , the method includes creating a first data structure with digital content metrics of one or more content items. The digital content metrics include an association to one or more user categories or user clusters for each content item. 
     In step  804 , it is determined if a search is carried out, i.e. if a search action is identified, for a content item among the one or more content items. If this is not the case, then the method proceeds to step  814 . 
     If a search action is determined in step  804 , then in step  806  discovery alert, such as an irregularity notice is generated. The discovery alert relates to metadata of the content item and/or the association to one or more user categories for the content item. 
     In step  808 , verification of the metadata and/or association in the discovery alert is performed and in step  810  it is determined if the verification was successful or unsuccessful. 
     If the verification was a success, i.e. it is verified that the metadata and association are correct and applied to a correct user category, i.e. user cluster and content item, then the method proceeds to step  814 . 
     If the verification was unsuccessful, thereby indicating that the metadata and/or association are either incorrect or incorrectly applied to a user cluster and/or content item, then in step  812  the digital content metrics for the content item is updated based on an outcome or result of the verification, i.e. to correct the metadata or association. 
     In step  814 , the digital content metrics for the one or more content items are extracted, including any updated digital content metrics from step  812 . 
     In steps  816  and  818 , ranking of the content items and subsequently promotion of the content item (as seen in  FIG. 7 ) takes place, respectively, using the extracted digital content items from step  814  to create a second data structure. 
     Referring back to  FIGS. 2, 3, and 4  the devices shown each include a processor (in other words a processing device)  202 ,  302  in communication with a memory  204 ,  304 . The memory  204 ,  304  of each of devices may comprise one or more of a main memory (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device), which communicate with each other via a bus. 
     The processing device  202 ,  302  of each of the devices may comprise one or more general-purpose processors such as a microprocessor, central processing unit, or the like. More particularly, the processing device may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processing device may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device is configured to execute the processing logic (instructions) for performing the operations and steps discussed herein. 
     The data storage device may include one or more machine-readable storage media (or more specifically one or more non-transitory computer-readable storage media) on which is stored one or more sets of instructions embodying any one or more of the methodologies or functions described herein. The instructions may also reside, completely or at least partially, within the main memory and/or within the processing device during execution thereof by the computer system, the main memory and the processing device also constituting computer-readable storage media. 
     The various methods described above may be implemented by a computer program. The computer program may include computer code arranged to instruct a computer to perform the functions of one or more of the various methods described above. The computer program and/or the code for performing such methods may be provided to an apparatus, such as a computer, on one or more computer readable media or, more generally, a computer program product. The computer readable media may be transitory or non-transitory. The one or more computer readable media could be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or a propagation medium for data transmission, for example for downloading the code over the Internet. Alternatively, the one or more computer readable media could take the form of one or more physical computer readable media such as semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disc, and an optical disk, such as a CD-ROM, CD-R/W or DVD. 
     In an implementation, the modules, components and other features described herein can be implemented as discrete components or integrated in the functionality of hardware components such as ASICS, FPGAs, DSPs or similar devices. 
     A “hardware component” is a tangible (e.g., non-transitory) physical component (e.g., a set of one or more processors) capable of performing certain operations and may be configured or arranged in a certain physical manner. A hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component may be or include a special-purpose processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. 
     Accordingly, the phrase “hardware component” should be understood to encompass a tangible entity that may be physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. 
     In addition, the modules and components can be implemented as firmware or functional circuitry within hardware devices. Further, the modules and components can be implemented in any combination of hardware devices and software components, or only in software (e.g., code stored or otherwise embodied in a machine-readable medium or in a transmission medium). 
     Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “receiving”, “determining”, “comparing”, “enabling”, “maintaining,” “identifying,” “generating,” “associating,” “inputting,” “authenticating,” “adding,” “transmitting,” “processing,” or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system&#39;s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. 
     It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. Although the present disclosure has been described with reference to specific example implementations, it will be recognized that the disclosure is not limited to the implementations described, but can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.