Patent Description:
The modem data-rich world is characterized not only by routine reliance on digital information sources, but also by a paradigm shift from a "pull model" of information retrieval, in which users explicitly search for information, to a "push model," in which relevant information is automatically recommended to them. Not surprisingly, therefore, large-scale online service providers, such as e-commerce websites and social networks, tend to integrate some form of "entity recommendation" into their service offerings as a core part of the user experience. Within a large system, serving relevant information in the right context - for example, recommending relevant products to a user out of hundreds of millions of products on an e-commerce website, or surfacing the right post that the user may want to consume in the current context out of millions of posts on a social media platform - constitutes a challenging computational problem.

The machine-learning community has devoted significant research to the entity-recommendation problem and developed multiple approaches to its solution. However, typical formulations described in the literature take a rather simplistic view of the whole system by modeling only two entities, such as the user and the item to be recommended. While such simple abstractions provide tractable formulations that can be addressed using rigorous algorithms, they ignore subtleties and complexities of key information available in the system. To capture the key information in the system, several practical recommendation systems use content filtering approaches, where joint features for (user, item) tuples are hand-designed and then fed into a classifier. Designing features is, however, a difficult problem, especially for real-world systems with several sources of information. Consequently, such methods often suffer from issues such as, e.g., poor accuracy or a difficult maintenance cycle. A need, therefore, exists for an approach to entity recommendation that automatically captures information subtleties and complexities.

<CIT> discloses a system and method for evaluating short messages. A list of messages, pre-filtered by keyword and geographic origin, are scored using a stack of filters having either general scoring parameters of flexible, user defined parameters. Each filter examines the messages for characteristics such as whether they have been re-sent, they contain particular abbreviations or repetitions, a minimum number of words, profanities or other characteristics indicative of their value to a user. A report consisting of the scored messages, i.e., the message and the score associated with the message after application of all relevant filters to the message, is then sent to a user.

<CIT> discloses intelligent message screening. Electronic messages are processed based on criteria relating to the sender, the content, and the personalization of the message. The content of the messages, the recipient's address book, and parameters such as desired keywords and undesired keywords are employed in determining a numeric ranking for each message. Based upon the ranking, messages are assigned to a category indicating an expected response of the recipient, such as read, reply, and save, or simply read. Messages in the lowest category are marked for deletion. Fuzzy logic is preferably applied in determining the category to which a message is assigned based on its ranking. Content importance, sender importance and degree of personalization are combined in a non-linear manner to rank a message. Based on the recipient's actual response to a message, the priority of subsequent similar messages is adjusted.

<CIT> discloses a method for generating a product recommendation in a retail system, which includes collecting a dataset containing a plurality of entities and attributes for the entities. Relationships between the plurality of entries are generated. The plurality of entities, attributes and relationships are stored in a knowledge graph. A representation of the plurality of entities, attributes and relationships stored in the knowledge graph is learned. Zero-shot learning is performed for a new entity. The new entity and attributes for the new entity are stored in the knowledge graph. A recommendation for a user is generated based on the knowledge graph.

<CIT> discloses a microprocessor executable method which transforms unstructured natural language texts by way of a preprocessing pipeline into a structured data representation of the entities described in the original text. The structured data representation is conducive to further processing by machine methods. The transformation process is learned by a machine learned model to identify relevant text segments and disregard irrelevant text segments. The resulting structured data representation is refined to more accurately represent the respective entities.

<CIT> discloses a computer system optimized for implementing a neural network nodal graph that has dense inputs and sparse inputs. The computer system has a local machine that receives user inputs and is optimized for computing power, and has a remote machine that stores embedding matrices and parameters, and is optimized for memory capacity. In accordance with a cost function applied to each node, the neural network nodal graph is divided into graph segments based on its types of inputs and needed computing resources for execution. In accordance with the cost functions, the graph segments are divided between the remote and local machines for execution, and the results of all the graph segments are combined in the local machine.

<CIT> discloses a networked computer system which identifies, optimizes and recommends content sources for users. The content sources are be used for providing news feeds or search results based on content contributed by such sources.

Described herein are aspects of data processing systems, methods, and computer program products (as embodied in machine-readable media) for entity recommendation that can capture subtleties and complexities not accounted for in simplistic two-entity models by allowing the items to be recommended and the recipients of the recommendation (e.g., users) to be modeled as "complex entities" composed of one or more static sub-entities and/or a dynamic component, and by utilizing information about multiple relationships between the sub-entities. The recipients of the recommendation are hereinafter also referred to as "first items," and the items to be recommended are hereinafter also referred to as "second items. " The first and second items are, respectively, users and messages recommended for their consumption.

The static sub-entities, hereinafter also simply referred to as "entities," are unique, fixed instances of, generally, multiple entity types. The first and second items are instances of respective first and second complex entity types. Each complex entity type is, in accordance herewith, defined by a corresponding set of associated (sub-)entity types (selected among the multiple entity types), sometimes in conjunction with a dynamic component that renders each complex-entity instance (i.e., each item) unique. For example, a publication item may be represented by its author (a static entity that may be shared among multiple publications), the journal in which it was published (another static entity that may be shared among multiple publications), and its text, or a unique portion of its text (e.g., title or abstract), constituting the dynamic component. As a special case corresponding to a "degenerate" complex entity type, a set of items may consist simply of entities of a single type. For example, user entities may be used as such to represent first or second items.

Relationships between entities (or, synonymously, "entity instances") are represented by graphs. Relationships between a pair of any two entity types are reflected in a bipartite graph. While the discussion is centered on bipartite graphs representing two different types of entities, the concepts can straightforwardly be extended to graphs between entities between entities of the same type. (One non-limiting way of doing so, within a framework of bipartite graphs, would be to represent the relationships among entities of a single entity type as a bipartite graph between two copies of the entities of that single type. ) Graphs (e.g., bipartite graphs) are used herein to inform the computation of recommendations of the second items to the first items; in a way, the first and second items "inherit" the relationships between their associated entities.

Given definitions of the first and second items and mathematical representations of their sub-entities and dynamic components (if any), and based on bipartite graphs representing relationships between the sub-entities, the approach described herein automatically scores pairs of a first item and a second item according to the relevance of the second item to the first, enabling recommendations to be made based on the scores, without the need for hand-designing features that capture the rich information reflected in the complex-entity definitions and graphs. In various embodiments, the scoring utilizes a multi-layer architecture. In the first layer, computational models for the bipartite graphs, e.g., graph-based convolutional neural networks (CNNs), are used to compute vector representations of the entity instances associated with the first and second items. In general, each item has multiple such associated vector representations (herein also "entity vector representations," to distinguish them from the overall representations of the items), which may correspond to different sub-entities and their respective associated bi-partite graphs, or to multiple different bipartite graphs involving the same sub-entity. In one or more subsequent layers, these multiple entity vector representations, along with any dynamic component, are combined into item vector representations (e.g., using a neural network to merge entity vector representations associated with different bipartite graphs between the same two entity types, and/or concatenating entity vector representations and dynamic-content representations). The resulting item vector representations for pairs of a first and a second item are, in the last layer, fed into a classifier (e.g., a multi-layer perceptron (MLP)) to generate relevance scores. Recommendations can then be generated by filtering, ranking, etc. based on the relevance scores.

The multi-layer architecture can be trained, in some embodiments end-to-end, with a suitable learning algorithm to optimize a specified performance metric based on labeled training data. Described herein is a metric - applicable, in particular, to imbalanced data - that does not only reward the occurrence of relevant items in the top-scoring items, but also penalizes the occurrence of irrelevant items in the top-scoring items. This metric can be optimized indirectly in an iterative process that involves, in each iteration, training the architecture to optimize a surrogate metric, and then using the trained architecture to score the training data, and selecting, based on the scores, a subset of the training data as the training dataset for the next iteration.

In one aspect, the recommendation approach described herein is applied to message recommendation within a conversation platform where messages are posted to channels and users have access to all channels with which they are associated. To avoid information overload to users in the face of large numbers of messages they can access, automatic message recommendation is used to surface only a small number of particularly relevant messages to a given user. These message recommendations may be computed in real-time as messages come in by first weaning the potentially large number of recipients based on recipient shortlists associated with the message author, then scoring users on the shortlist using a multi-layer architecture operating on complex entities and bipartite graphs as described above, and optionally further refining the subset of users identified based on the scoring using explicit user feedback.

The foregoing will be more readily understood from the following detailed description of various embodiments, in particular, when taken in conjunction with the accompanying drawings.

The present disclosure describes an approach to automated recommendation generation that is applicable to a wide variety of recommendation problems, entity types, and relationships between the entities. Entities may include, for instance, users (acting, e.g., as authors of publications, posts, or other content; senders or recipients of messages; buyers or sellers of products; etc.), documents (e.g., publications, posts, messages, media content, etc.), venues (e.g., journals, conferences, cities, communication channels, etc.), affiliations (e.g., institutions, social-network groups, etc.), products/services, and many others. Relationships between these entities may include, for example, connections or similarity between entities of the same type (e.g., social connections between users, citations between documents, similarity between products) and transaction or associations between entities of different types (e.g., purchase transactions of users for products, associations of documents with publication venues or authors, user affiliations with institutions, etc.). In the presently claimed invention the disclosed approach is used for determining recommendations of messages to users. A recommendation problem can be characterized by the nature of the recipients of the recommendation (i.e., the first items) and the kinds of items to be recommended (i.e., the second items), both of which are herein defined in terms of (static) entities and (usually, but not necessarily) a dynamic component. In the following, a system and neural-network architecture for generating recommendations of second items to first items from mathematical representations of the entities and dynamic components making up the items and of relationships between the entities are described with respect to the accompanying drawings. In the claimed invention the first items are users and the second items are messages.

<FIG> is a schematic block diagram of a data processing system for generating recommendations (hereinafter also referred to as a "recommendation system" <NUM>), in accordance with various embodiments. The recommendation system <NUM> can be implemented with a suitable combination of hardware and/or software, and generally includes one or more suitably configured or programmed hardware processors (such as central processing units (CPUs), graphic processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.). In various embodiments, the recommendation system <NUM> is implemented by a general-purpose computer (e.g., as illustrated in <FIG>) executing software, or a cluster of multiple such computers intercommunicating via a network. In brief, the computer or cluster includes one or more CPUs and/or GPUs, as well as (volatile and/or non-volatile) computer memory storing data and program code for execution by the CPU(s) and/or GPU(s). The computer may also include input/output devices (e.g., a keyboard/mouse and screen display for user interactions) and/or a network interface (e.g., for connecting to the internet).

The recommendation system <NUM> stores, in memory, item definitions <NUM> for the recommendation recipients and the items to be recommended (i.e., first and second items) in terms of sets of entity types and dynamic components, and mathematical input representations of the entities <NUM> and dynamic components <NUM> as well as of graphs <NUM> reflecting relationships between the entities <NUM>. In various embodiments, the input representations of the entities <NUM> and dynamic components <NUM> take the form of vectors, such as feature vectors derived from attributes of the entities, one-hot encodings (binary vectors reflecting, e.g., the numerical value of an entity ID within a dataset), bag-of-words vectors of text input, etc.; these and other suitable entity vector representations are well-known to those of ordinary skill in the art. The graphs are, in various embodiments, represented as adjacency matrices, i.e., matrices whose dimensions correspond to the number of instances of the respective entity types (the nodes of the graph) and whose cells reflect whether an entity of one type is connected with an entity of the other type and, optionally, how strong that connection is. The mathematical input representations of the entities <NUM>, dynamic components <NUM>, and graphs <NUM> may be provided directly as input to the system <NUM>, or may be computed by the system <NUM> from other input representations (e.g., from text constituting dynamic components, unique entity names, or datasets defining the entities in the form of text, images, etc.).

The recommendation system <NUM> further includes a number of functional blocks <NUM>, <NUM>, <NUM>, <NUM> that implement computational models and algorithms for processing the item definitions <NUM> and the mathematical input representations of the entities <NUM>, dynamic components <NUM>, and graphs <NUM> to generate recommendations. Specifically, graph-based computational models <NUM> created from the representations of the various graphs <NUM> (e.g., graph-based convolutional neural networks) operate, independently for each graph, on the input vector representations of the entities <NUM> (constituting the nodes) of the graph to compute respective entity vector representations <NUM> that capture the relationships reflected in the graph. Since the relationships between the entities are not affected by the item definitions, the entity vector representations <NUM> may be precomputed as well as shared between multiple recommendation problems to which they are relevant. Based on the item definitions <NUM> for a given recommendation problem, merging component <NUM> combines all entity vector representations <NUM> associated with each item, along with the input representation of any dynamic component <NUM> of the item, into an item vector representation <NUM>. Classifier model <NUM> can then score any pair of a first item and a second item, taking the associated item vector representations <NUM> as input. The resulting scores <NUM>, which quantify how relevant the second item is to the first item, may be used by recommendation generator <NUM> to generate recommendations, e.g., by identifying pairs of a first item and a second item, among a set of pairs, that have a relevance score <NUM> above a certain threshold, or by ranking the pairs based on their relevance scores <NUM> and returning a specified number of top-ranking pairs. The recommendations may be provided as output, e.g., to a user via a suitable user interface, or to a downstream computational process.

In some embodiments, the computation of item vector representations by the merging component <NUM> and scoring of pairs is triggered by the recommendation generator <NUM> and confined to items of interest at the time. For example, recommendations may be generated on demand for a given user (constituting a specific "first item") by scoring pairs of the item vector representation for that user and the item representations of all potentially relevant second items. Conversely, receipt of a new item to be recommended in a system (a new message) may trigger the computation of a corresponding item vector representation and scoring of the relevance of the item to all potential recommendation recipients.

For a more formal mathematical description of an algorithm for scoring pairs of first and second items in accordance with various embodiments, consider the following problem formulation: Let the first item (the recommendation recipient) be denoted by <IMG> (for "user," though the first item may also be a document or other type of item), and let the second item (the item to be recommended) be denoted by <IMG> (for "item"); the goal is to determine a score for the relevance of <IMG> to <IMG>. In general, both <IMG> and <IMG> are complex entities with one or more static sub-entities and (optionally) a dynamic component. Further, let ε(<NUM>),. , ε(E) be given entity types, and let ε(i)(j) denote thej-th entity of type i. For example, if ε(<NUM>) denotes a set of authors, then ε(<NUM>)(j) denotes thej-th author in the set. With these notations, <IMG> can be defined as: <MAT> where each user <IMG> has k<NUM> entity types <MAT> denotes the entity instance of type ε(ai)associated with <IMG>, and ζ(<IMG>) is the dynamic component of U. For example, a research paper <IMG> written for a conference C by author A may be defined in terms of a conference entity <MAT> and an author entity <MAT>, along with the dynamic component <MAT>, which may be, e.g., a <IMG>-dimensional word-to-vector embedding of the paper abstract and title. Similarly, the item <IMG> can be defined as: <MAT> where each item <IMG> has k<NUM> entity types ε(bi)(i ∈ [k<NUM>]), <MAT> denotes the entity instance of type ε(bi) associated with <IMG>, and <MAT> denotes the <IMG>-dimensional dynamic component of the item. The entity types defining the item <IMG> may be different from or the same as those defining the user <IMG>, depending on the type of problem. For instance, citation items, like the papers to which they are recommended, may be defined in terms of authors, conference venues, and word-to-vector embeddings of the title and abstract of the paper to be cited.

Relationships between the static entities are represented by bipartite graphs (generally multiple graphs for multiple relationships) whose nodes correspond to entities and whose edges correspond to the relationships between any two entities. For example, the edges of an author-conference graph may represent the number of times an author has published at a certain conference in the past. Let the bipartite graph between entities of types ε(a) (which are |εa| in number) and ε(b) (which are |εb| in number) be denoted by: <MAT> where Va,b is the set of (|εa|+|εb|) nodes and <MAT> is the adjacency matrix of the graph, the rows and columns of Aa,b being associated with entity instances of types ε(a) and ε(b), respectively. For any two entity types associated with entity instances of types ε(a) and ε(b), there may be multiple graphs representing different relationships. Thus, let
<MAT>
be the set of graphs between entities ε(a) and ε(b). Further, let the set of all existing graphs for any pair of entities within a set {ε(<NUM>),. , ε(E)} be denoted by
<MAT>.

<FIG> is a diagram conceptually illustrating an example collection of entities and relationships therebetween, in accordance with various embodiments. Three types of entities, ε(a), ε(b), and ε(c) are depicted as squares, circles, and triangles, respectively. The number of entity instances may differ between entity types, as shown. Relationships between pairs of entities are reflected in a number of (five, in this example) bipartite graphs <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. Bipartite graphs <NUM>, <NUM>, <NUM>, <NUM> capture relationships between two different types of entities, with bipartite graphs <NUM>, <NUM> corresponding to two different relationships between the same pair of entities. Bipartite graph <NUM> represents the relationships between the entities of a single type, which are duplicated to fit the relationships into the framework of bipartite graphs. To illustrate by way of example, assume that the squares, circles, and triangles represent researchers ε(a), fields of research ε(b), and scientific journals ε(c), respectively. Bipartite graph <NUM> may reflect the fields of research in which the various researchers work, and bipartite graph <NUM> may represent which journals include publications in the various fields of research. Bipartite graphs <NUM>, <NUM> may capture two different relationships between researchers and journals, such as which journals a researcher has published in, or for which journals a researcher has previously served as a reviewer. The bipartite graph <NUM> may represent relationships between the researchers themselves, such as co-authorship of publications. Such rich information about the relationships between the three entities may be useful, for instance, to generate recommendations of possible journals <IMG> in which to publish a research article <IMG>, where the article is represented by the researcher <MAT> that authored it, optionally the field of research <MAT>, and a dynamic component based on the text, and where the possible journals <IMG> are represented by the journal entities <MAT> themselves.

Turning now to an example algorithm for computing recommendations in accordance with various embodiments, <FIG> is a schematic diagram illustrating a multi-layer neural-network architecture <NUM> suitable for this purpose. In a first layer <NUM>, computational models <NUM> of various bipartite graphs <NUM>, <NUM> are used to compute entity vector representations <NUM>, <NUM>, <NUM>, <NUM> (corresponding to <NUM> in <FIG>), herein also "entity embeddings," for the entity instances that constitute the nodes of the graphs <NUM>, <NUM>. (The term "embedding" is herein understood, consistently with its general use in the context of neural networks, as a learned, continuous vector representation (i.e., a vector of continuous numbers) of a discrete or categorical variable. ) In the depicted (non-limiting) example, bipartite graphs <NUM> represent relationships between entities of types ε(a) and ε(b), and bipartite graphs <NUM> represent relationships between entities of types ε(a) and ε(c); of course, the architecture <NUM> may similarly incorporate further entities and further relationships between entities of different types or of the same type.

In various embodiments, the computational models for the bipartite graphs are, or include, graph-based CNNs as known to those of ordinary skill in the art. For a given graph G = (V, A), a convolution on a graph signal <MAT>, with a filter cw parameterized by <MAT> in the Fourier domain, is defined as follows: <MAT> where U is the eigenvectors matrix of the normalized Laplacian L of graph G. To reduce computational expense (which is otherwise high due to <IMG>(|V|<NUM>) multiplicative complexity), the filter cw(Λ), which is a function of eigenvalues A of L, may be approximated in terms of Chebyshev polynomials: <MAT> where <MAT>, λmax is the largest eigenvalue of L, I|V| is an identity matrix of size |V|, Wh are coefficients to be learned during training of the CNN, and Th are the h-th order Chebyshev polynomials defined recursively as Th(x) = 2xTh-<NUM>(x) - Th-<NUM>(x) with T<NUM>(x) = x and T<NUM>(x) = <NUM>. Applying this approximation to the convolution operation, the convolution on the graph signal can be computed more simply as: <MAT> where <MAT>. Since this approximation is a function of the H-th order Laplacian, it performs local operations by considering only the H-th order neighborhood of a node. These local operations are then applied across all locations in the graph.

Returning to the application of CNNs to a bipartite graph Ga,b representing relationships between entities of types ε(a) and ε(b), let the graph, for simplicity of exposition, be represented as G = {Va ∪ Vb,A}, where Va and Vb correspond to entity instances of types ε(a) and ε(b), respectively, and A is the adjacency matrix between the nodes defined as: <MAT> The normalized Laplacian for the graph is defined as: <MAT> where D is the diagonal degree matrix of A and I is the identity matrix of size |εa|+|εb|. Further, L̃ is defined as above.

The order H of the polynomial approximation can, in principle, be determined by cross-validation on training data, although this process is computationally expensive. It has been shown both theoretically and empirically that, for large values of H, all entity embeddings converge to the same vector during the optimization routine, deteriorating the classification results. In various embodiments, therefore, a small value, e.g., H = <NUM> is chosen. Heuristically, the graph convolution operation is a form of message passing with messages/features being transformed across edges of the graph. For bipartite graphs, it is desirable to get the same entity embedding for multiple entity instances of type ε(a) that interact with the same entity instances of type ε(b) with similar strength (as reflected in the edge weights) and vice versa; this is the phenomenon of collaborative filtering. Choosing an order-<NUM> approximation of the convolution operation implies that messages are being passed from the set of nodes for ε(a) to the set of nodes for ε(b). With H = <NUM>, the convolution operation reduces to: <MAT> In some embodiments, the number of parameters in the network is further reduced, to avoid overfitting and numerical instabilities, by constraining W<NUM> = W<NUM> + W<NUM>, resulting in: <MAT>.

Given the above convolution operation along with a graph Ga,b between two types of entities ε(a) and ε(b) and its normalized Laplacian L̃a,b, DCN-dimensional embeddings of entity instances can be computed by applying DCN convolutional filters cw to the graph. Let <MAT> be the input feature matrix for entity instances of type ε(a) (i.e., a matrix composed of p-dimensional input vector representations <NUM> of all entity instances of that type), and let <MAT> be the input feature matrix (composed of q-dimensional input vector representations) of the entity instances of type ε(b). Using the graph-convolutional operation then yields: <MAT> where RELU := max(<NUM>, a), and <MAT> and <MAT> are the feature matrices obtained by applying the convolutional operation, which are composed of DCN-dimensional entity vector representations <NUM> for the entities associated with ε(a) and ε(b), respectively. Each column of <MAT> corresponds to one convolutional filter of Ga,b. Thus, the layer <NUM> in the architecture <NUM> generates, in the described embodiment, graph-convolutional second-order polynomial ("G2-P2") embeddings (contained in the matrices Z(a), Z(b)) for the entity instances associated with graph Ga,b.

As mentioned, there can be multiple graphs between entity types ε(a) and ε(b). For example, relationships such as the "number of papers published" or the "number of papers cited" in a conference by an author may be represented by two different graphs. In layer <NUM> of the architecture <NUM> (herein also the "embedding merge layer"), the (G2-P2) entity embeddings for a given first entity type that emerge from relationships with a given second entity type (which may, in special cases, be the same entity type as the first) are merged across all those relationships (i.e., across all graphs between the first and second entities). For example, with reference to <FIG>, all entity embeddings <NUM> of entities in ε(a) with regard to ε(b) are merged into merged entity embeddings <NUM>; all entity embeddings <NUM> of entities in ε(b) with regard to ε(a) are merged into merged entity embeddings <NUM>; all entity embeddings <NUM> of entities in ε(a) with regard to ε(c) are merged into merged entity embeddings <NUM>; and all entity embeddings <NUM> of entities in ε(c) with regard to ε(a) are merged into merged entity embeddings <NUM>.

The merging can be accomplished, e.g., by first concatenating the entity embeddings of the first type with regard to the second type, and then merging them using a fully connected neural network layer. Formally, with <IMG> denoting the set of graphs between entity types ε(a) and ε(b), the entity embeddings for entity instances of ε(a) computed with regard to those of ε(b) are merged, and the same is done for the entity instance of ε(b) with regard to those of ε(a). Written, jointly for all entity instances of ε(a) and ε(b), in the form of the feature matrices, the merging can be expressed as: <MAT> where <MAT>, with DMN being the embedding dimension of the embedding-merge layer <NUM>, represents the collection of merged embeddings of all entities in ε(a) with regard to ε(b) and of all entities in ε(a) with regard to ε(b), The W(a), W(b) ∈ <MAT> and <MAT> are the weights and biases of the embedding-merge layer <NUM>. Note that different values of DMN may be chosen for different entity types.

Having merged entity embeddings across graphs between a given pair of entity types, merged entity embeddings are combined across the sub-entities of an item and augmented, if applicable, with an embedding of the dynamic component <NUM> of the item, in layer <NUM> (herein also the "static-dynamic layer"), to create item vector representations <NUM>. The combination can, for example, take the form of a concatenation of the respective vector representations. Thus, the first and second items may be represented as: <MAT> and <MAT> where <MAT> are the merged embeddings of <IMG> or <IMG>, respectively, from all graphs between ε(i) and ε(j), and where, if an item does not include instances of ε(i) as sub-entities or there is no relationship between ε(i) and ε(j), the corresponding <MAT> or <MAT> are taken to be zero. For example, there may be user-author graphs and user-conference graphs, but not author-conference graphs; in that case, the merged embeddings for author-conference graphs will be zero. The layers <NUM>, <NUM> collectively implement an embodiment of the merging component <NUM>.

The item vector representations Φ(<IMG>) and Φ(<IMG>), e.g., in the form of a concatenated vector [Φ(<IMG>); Φ(<IMG>)] (herein also the "static-dynamic content embedding" of the pair of first and second items), are provided as input to a classifier neural network <NUM>. The classifier neural network <NUM> may be, e.g., an MLP that computes a relevance score for the pair of first and second items according to: <MAT> where <MAT> are the network weights in the MLP layers, and σ is the sigmoid function (for a binary classification problem).

<FIG> summarizes, in the form of a flow chart, a method <NUM> of generating recommendations in accordance with various embodiments. The method <NUM> may be performed by a recommendation system <NUM> as shown in <FIG>, which may implement, e.g., the neural-network architecture <NUM> shown in <FIG>. The method <NUM> involves receiving, as input, graph data for bipartite graphs representing relationships between entity instances of multiple entity types, and storing representations of the graphs and entity instances in memory (act <NUM>). Further, the method <NUM> includes operating on the stored representations, using computational models <NUM> for the bipartite graphs to compute entity vector representations for the entity instances (e.g., entity embeddings Z(a), Z(b) as defined above for a graph Ga,b) (act <NUM>). In various embodiments, these computational models are or include graph-based CNNs, although other types of models, such as, e.g., graph-based recursive neural networks (GRNNs) or meta-path-based models (e.g., metapath2vec) may also be used.

Based on definitions of first and second items (i.e., items to receive recommendations and items to be recommended) in terms of their constituent static sub-entities and/or dynamic components, the entity vector representations are combined across bipartite graphs and, if applicable, with representations of the dynamic components, in act <NUM> (performed by merging component <NUM>), to generate item vector representations for the first and second items. In some embodiments, this act involves merging entity vector representations computed for instances of a given first entity type relative to instances of a given second entity type across multiple bipartite graphs reflecting multiple respective relationships between the pair of entity types, e.g., using an embedding-merge layer <NUM> that may include a fully connected neural network. Further, for items that include multiple static sub-entities and/or a dynamic component in addition to the static sub-entity or - entities, the merged entity vector representations and/or dynamic component may be concatenated, e.g., in static-dynamic layer <NUM>. Alternatively, entity vector representations may be combined with each other and/or with dynamic components in other ways, e.g., using outer products. Relevance scores for pairs of a first item and a second item are then computed based on the corresponding item vector representations (act <NUM>), using a classifier model <NUM>,<NUM>, such as an MLP, recursive neural network (RNN) or other neural-network model, kernel support vector machine or other type of classifier. Once the pairs of items have been scored according to relevance of the second item to the first item, recommendations are generated based on the relevance scores and output (act <NUM>), e.g., to a user or another computational component. Generating the recommendations may, for example, involve ranking and/or filtering based on the scores.

Depending on the particular application, acts <NUM>-<NUM> may be precomputed or computed (e.g., in real-time) in response to some trigger event. For example, in some cases, entity vector representations for the static entities are precomputed based on the bipartite graphs and stored for later use in response to a trigger event such as, e.g., a request for recommendations to a given first item (a certain user) or receipt of a new second item (a new message). In response to a recommendation request by a user, for instance, item vector representations for that user and for some or all second items (a subset of second items resulting from some preliminary filtering) may be created and scored, in pairs, according to relevance. Similarly, addition of a new item to be recommended may trigger the computation of item vector representations for that item and at least a subset of first items for a new message, a subset of users potentially interested in the message) and scoring of the first items according to relevance of the second item to the first items. In some embodiments, the computation of entity vector representations based on the bipartite graphs is likewise performed upon a trigger event. For example, upon sign-up of a new user to a social network, graphs associated with users and/or other entities (e.g., publications authored by the users) may be updated to include the new users, and entity vector representations may then be computed (or recomputed) from the updated graphs.

The computational models and algorithms used in generating recommendations in accordance herewith, such as the neural-network architecture <NUM>, may be trained based on labeled training data: <MAT> where yi is the label of the i-th pair of first and second items in the dataset, which can be, for instance, any integer number between <NUM> and C for a multi-class problem (as which many real-world problems can be cast). (Alternatively, to allow for comparisons on various dimensions, the labels yi could be cast as vectors of C real numbers. ) Given D, entities ε(<NUM>),. , ε(E), and graphs <IMG>, the goal of training is to find a scoring function s(<IMG>,<IMG>) that works best, in accordance with a certain metric <IMG>, for computing relevance scores for pairs (<IMG>). For example, for binary or real-valued labels yi, the goal may be that the computed scores are consistent with the training data, such that ( s(<IMG>) > s(<IMG>) if yi > yj.

For some recommendation problems, the performance of the described approach can be adequately evaluated with conventional metrics such as, e.g., the area under the curve (AUC), normalized discounted cumulative gain (NDCG), precision@k, or other metrics known to those of ordinary skill in the art. Many important recommendation problems, however, are characterized by imbalanced data, where the number of irrelevant items far exceeds the number of relevant items (where the notion of relevance may, e.g., account for a cut-off for the number of items that are to be recommended at a given time). In the context of a message recommendation system, for instance, the number of messages that users desire to have surfaced within a given time period (e.g., a day) may be merely a small fraction of the number of new messages submitted to the system during that time. In this and similar circumstances, it is important that the k top-scoring items (hereinafter the "top-k" items), which are the items that will be recommended, not only include highly relevant items, but also avoid irrelevant items (since irrelevant items would needlessly occupy space otherwise available to the next-most relevant items). AUC does not capture this aspect.

In accordance with various embodiments, therefore, a new metric, hereinafter termed "AUC-Rel@k," is used to optimize and/or evaluate the performance of the recommendation algorithm. To define this metric, consider a set of n labeled data points (x<NUM>,y<NUM>),. , (xn,yn), where xi ∈ X denotes pairs (<IMG>) of first and second items, and yi ∈ {<NUM>,<NUM>} is a binary label indicating whether a pair is relevant (<NUM>) or irrelevant (<NUM>). Let n+ and n- be the numbers of positively labeled (<NUM>) and negatively labeled (<NUM>) data points. Further, consider a scoring function <MAT>, and let γs be the permutation that sorts data points according to the scores given by s and (xγs(i), yγs(i)) be the data points as resorted by γs, such that s(xγs(i)) ≥ s(xγs(j)) if i ≥ j. With these notations, define: <MAT> <MAT> where S(s, i) denotes the set of items j that are scored higher than item i, and Tk,n (s, i) is an indicator function which is one when item i is in the top-k items ordered by score s. Furthermore, define: <MAT> <MAT> where S+ (s, i) denotes the set of positively labeled items j that are scored higher than positively labeled item i, and T+k,n(s,i) is an indicator function which is one when item i is in the top-min(k, n+) (i.e., the top-k or the top-n+, whichever is smaller) positively labeled items ordered by score s.

Now, suppose that the number of positives and negatives in the top-k elements (sorted by s) are k+ and k-, respectively. Then, AUC - Rel@k is defined as: <MAT> where <IMG> is an indicator function that is one when the condition in square brackets is met. This metric considers pairwise comparisons between irrelevant items in the top-k items vs. the top-min (k, n+) relevant items (sorted by score s), and computes how many times a relevant item has secured a higher score than an irrelevant item in the pairwise comparison. Thus, AUC - Rel@k rewards the presence of every relevant item in the top-k items, and penalizes the presence of irrelevant items in the top-k items over relevant items outside the top-k. Note that AUC - Rel@k is upper-bounded by AUC@k (that is, the AUC computed only for the top-k items), as AUC - Rel@k considers more pairwise comparisons than AUC@k, namely comparisons of relevant items beyond the top-k with irrelevant items in the top-k, which are always lost (i.e., amount to zero) while being counted towards the normalization (i.e., the denominator). It can be shown that: <MAT>.

For multiple first items <MAT>, a Micro version can be defined (in analogy to the conventional AUC@K) as follows: <MAT> where AUC - <IMG> is AUC - Rel@k computed for items recommended to <IMG>. Similarly, a Macro version can be defined as follows: <MAT> where the notations have the same meaning as defined above except that they are now corresponding to <IMG>.

<FIG> is a schematic block diagram of an example data processing system, in accordance with various embodiments, for training neural-network components of a recommendation system <NUM> (e.g., neural-network architecture <NUM>) (hereinafter also "training system" <NUM>). Like the recommendation system <NUM>, the training system <NUM> can be implemented with a suitable combination of hardware and/or software, such as, e.g., with a general-purpose computer (e.g., as illustrated in <FIG>), or cluster of computers, storing suitable software in computer memory and executing the software with one or more CPUs or GPUs, or with one more special-purpose processors such DSPs, FPGAs, or ACICs. The training system <NUM> serves to train a neural-network architecture <NUM> implementing the computational models <NUM>, merging component <NUM>, and classifier models <NUM> by optimizing adjustable network parameters <NUM> of the neural-network architecture <NUM> for a given metric <NUM>, which may be specified as input to the training system <NUM>. For the neural-network architecture <NUM> described above with respect to <FIG>, for example, these adjustable parameters <NUM> may include the convolutional filter parameters <MAT> (which include the coefficients Wh of the Chebyshev approximation for all DCN filters), the weights <MAT> and biases <MAT> of the embedding merge layer <NUM>, and the weights <MAT>, <MAT> of the MLP (classifier neural network <NUM>) that performs the classification. In some embodiments, the neural-network architecture <NUM> is trained end-to-end, that is, all network parameters <NUM> are adjusted jointly in a single process (rather than separate processes for different neural-network components).

Optimization of the parameters <NUM> involves executing a learning algorithm <NUM> to iteratively optimize a surrogate metric <NUM> for the performance metric <NUM> based on labeled training data <NUM> provided as input to the training system <NUM>. The learning algorithm <NUM> may utilize, e.g., backpropagation of errors, which is well-known to those of ordinary skill in the art of machine learning, or any other suitable technique. Conventional performance metrics such as AUC or accuracy can be directly optimized by standard surrogate metrics <NUM> such as, e.g., cross entropy. This relation does, however, not hold for the above-defined metric AUC - Rel@k, which looks at partial data rather than taking a holistic view of the data as standard surrogate metrics do. In other words, optimizing, e.g., cross-entropy will not, in general, result in optimal AUC - Rel@k. In various embodiments, therefore, as illustrated in <FIG>, an additional optimization layer is added: Following execution of the learning algorithm <NUM> with the initial set of training data <NUM>, all pairs of first and second items in the training data <NUM> are scored using the trained neural-network architecture <NUM>, and the scoring is then used to select (as detailed below) a suitable subset <NUM> of the training data for another round of networking training using the learning algorithm <NUM>. This process is repeated iteratively, e.g., until a specified number of iterations have been completed. An evaluation component <NUM> can then quantify the performance of the neural-network architecture <NUM> thus trained, using AUC - Rel@k as the performance metric <NUM>.

Since the recommendation and training systems <NUM>, <NUM> both use the neural-network architecture <NUM>, they may, in some embodiments, be integrated with each other and share resources. In other embodiments, the training system <NUM> and the recommendation system <NUM> are separate and distinct entities, e.g., where only the training system <NUM> implements the learning algorithm <NUM> and only the recommendation system <NUM> includes the recommendation generator <NUM>. Both of these entities may include respective copies of the neural-network architecture <NUM>, and the network parameters <NUM> determined for the copy within the training system <NUM> may, once optimized, be communicated to the recommendation system <NUM> for use with its copy of the neural network architecture <NUM>. Alternatively, the training system <NUM> need not include a copy of the neural-network architecture <NUM> itself, but may instead exchange data with the recommendation system <NUM> during the training process to directly adjust the network parameters <NUM> within the recommendation system <NUM>, and once the parameters <NUM> have been determined, these communications may cease.

<FIG> is a flow chart illustrating, in more detail, an example method <NUM> of training a neural-network architecture <NUM> for computing recommendations to optimize for AUC - Rel@k, in accordance with various embodiments. The method <NUM> takes training data, test data, and a specified number of iterations as input, and returns a value of AUC - Rel@k as output. After receiving the input (in act <NUM>) and initializing the neural-network parameters <NUM>, the training dataset, and an iteration counter (in act <NUM>), the method <NUM> enters into an iterative loop that proceeds as long as the iteration counter is below the specified number of iterations (as determined at <NUM>).

In each iteration, the neural-network architecture <NUM> is trained on the current training dataset, in act <NUM>, to minimize an objective function such as, e.g., the regularized form of the negative log likelihood (corresponding to cross entropy) of the predicted score s(<IMG>) : <MAT> where si = s(<IMG>) are the scores computed by the neural-network architecture <NUM>, yi are the labels given to datapoint (<IMG>) in the training data, τ is the regularizer, and the second term is a sum of l<NUM>-norms of all the weight matrices in the architecture <NUM>. In some embodiments, the weights and subsequent embeddings from multiple bipartite graphs are learned using an Adam optimization algorithm (which is an extension to the stochastic gradient descent technique introduced by <NPL>(an opensource software library for dataflow and differentiable programming that is suitable for various machine-learning applications) with mini-batches. The trained neural-network architecture <NUM> is used to score all datapoints in the received training data (act <NUM>), and, based on the scores, a subset of the received training data is then selected to serve as the training dataset in the next iteration (act <NUM>), and the iteration counter is incremented (act <NUM>). (In the first iteration, where the neural-network architecture <NUM> is trained on the full set of received training data, the scoring act <NUM> may be superfluous. In subsequent iterations, where training is generally based only on a subset of the received training data, there will be datapoints (<IMG>) that have not been scored in the course of the immediately preceding network training. ) More specifically, the dataset for the next iteration includes, in accordance with some embodiments, the "top rel-k" datapoints per first item (e.g., user) <IMG>, aggregated across all first items, where "top rel-k" denotes the union of the top-k scored points and up to k positively labeled datapoints that are outside the top-k scored points (for a total of at most <NUM> datapoints, and fewer datapoints if there are fewer than k positive labeled points outside the top-k. ) When the iteration counter has reached the specified number of iterations, training is deemed complete, and the metric AUC - Rel@k is evaluated on the test data and provided as output (act <NUM>). The method <NUM> indirectly optimizes AUC - Rel@k. As will be appreciated by those of ordinary skill in the art, similar metrics for imbalanced data that likewise penalize irrelevant items in the top-k as well as rewarding relevant items in the top-k may also be optimized using the described process.

The recommendation approach described above with reference to <FIG> can be applied to many different problems and entities, such as to recommend relevant research papers for potential citation in a given paper, to recommend groups within a social network or media-sharing platform for users to post images and the like, or to recommend messages to users within a shared enterprise conversation platform, to name just a few.

To provide a specific example, <FIG> and <FIG> illustrate a system and method for message recommendation in an enterprise conversation platform (hereinafter also the "message system") such as, for example and without limitation, the Microsoft Teams ® product (by Microsoft Corporation, Redmond, WA). Within such a message system, users may each be part of one or more teams, and the teams may build and maintain conversation threads of messages exchanged between the team members, where messages are posted to message channels associated with the teams. In contrast to email communications, which are sent only to explicitly listed recipients, the messages within the channels may be accessible to all members of the associated team (or teams) as a means of facilitating conversation among large teams. Such unrestricted access can, however, lead to an excessively large number of messages that a given user can read, creating significant information overload for the user. As will be appreciated, an accurate recommendation system capable of identifying and surfacing the most relevant messages to the user can address this issue, allowing the user to ignore many (even the vast majority of) messages without risk of missing messages of particular importance to him. As will also be apparent, the AUC - Rel@k metric introduced above, due to its suitability for imbalanced data, provides a good measure of performance for such a system.

The recommendation problem for the described scenario can be stated as learning a function s(<IMG>) that, for a given user <IMG> (the first item) and a given message<IMG> (the second item), predicts whether the user would be interested in the message. In the training data, the label y for any pair (<IMG>) may be a binary indicator of whether the user has "engaged" with the message, where "engaging" may be defined, e.g., as "liking" or replying to a message. Consider a recommendation system that has three entity types: users <IMG>, authors <IMG>, and channels <IMG>. (Note that sets of users <IMG> and authors<IMG> generally include the same people (or at least overlap), but reflect different roles a user can take: that of recipient of the recommendation and that of author associated with a message. ) Given these entities, a user is defined simply by the user entity type: <IMG> = (<IMG>); and a message item is defined by the author and channel entity types and a dynamic component v(<IMG>) representing the content of the message: <IMG> = (<IMG>). Further, consider relationships between the entities that are reflected in one or more graphs between three pairs of entities: (<NUM>) user-user graphs, with unidirectional edges such as the number of common channels between users, the number of common conversations between users, etc.; (<NUM>) user-author graphs, with directional edges such as the number of times a user liked an author's post, replied to an author's post, etc.; and (<NUM>) user-channel graphs, with edges capturing interactions such as how many times a user visited a channel in the past month, how many messages a user posted on the channel, etc..

In principle, the above-described recommendation approach (e.g., as implemented using neural-network architecture <NUM>) can be used to score users for a given incoming message to determine the relevance of the message to each user. In practice, however, this application presents challenges. For one thing, the message system may provide state-free message notifications, such that, as soon as a new message comes in, a decision is made as to which users are to be notified of the message. Further, the number of users who can access the channel, and for whom a decision whether or not to surface the message is therefore to be made, can be very large; a popular message channel may, for instance, have thousands of users enrolled. Invoking the scoring function s(<IMG>) potentially thousands of times per metric in real-time may not be feasible with the available computational resources.

<FIG> is a block diagram of an example data processing system, in accordance with various embodiments, for generating message recommendations (hereinafter also "message recommendation system" <NUM>). Like the recommendation system <NUM>, the message recommendation system <NUM> can be implemented with a suitable combination of hardware and/or software, such as, e.g., with a general-purpose computer (e.g., as illustrated in <FIG>), or cluster of computers, storing suitable software in computer memory and executing the software with one or more CPUs or GPUs, or with one or more special-purpose processors such DSPs, FPGAs, or ACICs. The message recommendation system <NUM> may form part of a larger message system that includes user interfaces for users to enroll with message channels, post and read messages, etc., and that may keep message records and track user engagement with messages. The system <NUM> takes messages <NUM> and associated metadata, such as the author and message channel, as input and identifies users to whom the messages should be recommended, e.g., by surfacing them in a notification pane <NUM>. To use the scoring function s(<IMG>) in a manner that is computationally tractable, the system <NUM> integrates the computation of s(<IMG>) into a layered architecture designed to reduce computational cost.

In a first layer <NUM> (herein also "filtering layer"), the initially often large set of users with access to the channel to which a given message was posted is filtered based on an author-specific shortlist <NUM> of (e.g., a few tens of) users, reducing the pipeline cost by orders of magnitude. The shortlist <NUM> for a given author lists users to whom messages from that author are potentially relevant, as may be determined, independently from any message content, based on, e.g., the static relationships between users, authors, and channels. In some embodiments, the shortlist is compiled offline based on static user-author features with a learned standard classifier operating on top. To enable fast operation of the filtering layer <NUM>, the shortlists <NUM> may be precomputed and used for simple lookups. For a message from a particular author, the scoring function s(<IMG>) is then invoked, in the second layer <NUM> (herein also the "scoring layer"), only for users on the shortlist <NUM>. The scoring layer <NUM> may implement, e.g., components of the neural-network architecture <NUM>. In various embodiments, to speed up computation, the scoring layer <NUM> accesses precomputed entity vector representations <NUM> for the author, channel, and user, and concatenates them, along with the dynamic component, to create item vector representations to be provided as input to the classifier (corresponding to scoring function s). A third, optional layer <NUM> (herein also "post-hoc layer") achieves a personalized delivery model by taking explicit user feedback <NUM> (e.g., collected over time) into account when creating the final set of users that will be notified of the message, e.g., by adding message back or removing messages from the set of messages deemed relevant. The computed recommendations <NUM> can be sent to the notification pane <NUM> for output to the respective users.

<FIG> is a flow chart of an example message recommendation method <NUM> (e.g., as performed by the message recommendation system <NUM>), in accordance with various embodiments. The method <NUM> involves storing precomputed entity vector representations for users, authors, and channels (act <NUM>). Upon receipt of a message <NUM>, users with access to a channel associated with the message are identified (act <NUM>), and the identified users are then filtered, e.g., based on a user shortlist <NUM> associated with the author of the message, to create a set of users that, in general, includes a much smaller number of users than have access to the message (act <NUM>). The users within this smaller set are scored according to the relevance of the message to each of the users (act <NUM>). The scoring may involve computing item vector representations for the message and for all users within the set, and applying a classifier model to the item vector representations (e.g., as shown in <FIG>). A generally even smaller subset of the scored users is then determined based on the scoring (act <NUM>), e.g., by retaining only users for whom the score exceeds a certain threshold value. The threshold value may be user-dependent, and may set empirically to achieve, at least on average, a desired number of messages that will be surfaced to the user in the notification pane. Post-hoc logic may be applied to the subset of users to add and/or remove users (act <NUM>). For example, if, the user desires to see fewer or more messages from a particular author or from a particular channel, then we can remove such messages from, or add such messages to, our sorted list of messages. Other examples of suitable post-hoc logic to refine the message recommendations will occur to those of ordinary skill in the art. Finally, the message is recommended to all users remaining within the refined subset (act <NUM>).

In general, the operations, algorithms, and methods described herein may be implemented in any suitable combination of software, hardware, and/or firmware, and the provided functionality may be grouped into a number of components, modules, or mechanisms. Modules and components can constitute either software components (e.g., code embodied on a non-transitory machine-readable medium) or hardware-implemented components. A hardware-implemented component is a tangible unit capable of performing certain operations and can be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client, or server computer system) or one or more processors can be configured by software (e.g., an application or application portion) as a hardware-implemented component that operates to perform certain operations as described herein.

In various embodiments, a hardware-implemented component can be implemented mechanically or electronically. For example, a hardware-implemented component can comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented component can also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware-implemented component mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) can be driven by cost and time considerations.

Accordingly, the term "hardware-implemented component" should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily or transitorily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware-implemented components are temporarily configured (e.g., programmed), each of the hardware-implemented components need not be configured or instantiated at any one instance in time. For example, where the hardware-implemented components comprise a general-purpose processor configured using software, the general-purpose processor can be configured as respective different hardware-implemented components at different times. Software can accordingly configure a processor, for example, to constitute a particular hardware-implemented component at one instance of time and to constitute a different hardware-implemented component at a different instance of time.

Hardware-implemented components can provide information to, and receive information from, other hardware-implemented components. Accordingly, the described hardware-implemented components can be regarded as being communicatively coupled. Where multiple such hardware-implemented components exist contemporaneously, communications can be achieved through signal transmission (e.g., over appropriate circuits and buses that connect the hardware-implemented components). In embodiments in which multiple hardware-implemented components are configured or instantiated at different times, communications between such hardware-implemented components can be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware-implemented components have access. For example, one hardware-implemented component can perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware-implemented component can then, at a later time, access the memory device to retrieve and process the stored output. Hardware-implemented components can also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions. The components referred to herein can, in some example embodiments, comprise processor-implemented components.

Similarly, the methods described herein can be at least partially processor-implemented. For example, at least some of the operations of a method can be performed by one of processors or processor-implemented components. The performance of certain of the operations can be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors can be located in a single location (e.g., within an office environment, or a server farm), while in other embodiments the processors can be distributed across a number of locations.

The one or more processors can also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)).

Example embodiments can be implemented in digital electronic circuitry, in computer hardware, firmware, or software, or in combinations of them. Example embodiments can be implemented using a computer program product, e.g., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable medium for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers.

A computer program can be written in any form of description language, including compiled or interpreted languages, and it can be deployed in any form, including as a standalone program or as a component, subroutine, or other unit suitable for use in a computing environment.

In example embodiments, operations can be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations can also be performed by, and apparatus of example embodiments can be implemented as, special purpose logic circuitry, e.g., an FPGA or an ASIC.

In embodiments deploying a programmable computing system, it will be appreciated that both hardware and software architectures merit consideration. Specifically, it will be appreciated that the choice of whether to implement certain functionality in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or a combination of permanently and temporarily configured hardware can be a design choice. Below are set out hardware (e.g., machine) and software architectures that can be deployed, in various example embodiments.

<FIG> is a block diagram of an example computing system as may be used to implement the systems of <FIG>, <FIG>, and <FIG>, in accordance with various embodiments. In alternative embodiments, the computing system may be a machine operating as a standalone device or, alternatively, connected (e.g., networked) to other machines. In a networked deployment, the machine can operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a web appliance, a network router, switch, or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system <NUM> includes a processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory <NUM>, and a static memory <NUM>, which communicate with each other via a bus <NUM>. The computer system <NUM> can further include a video display <NUM> (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system <NUM> also includes an alphanumeric input device <NUM> (e.g., a keyboard or a touch-sensitive display screen), a user interface (UI) navigation (or cursor control) device <NUM> (e.g., a mouse), a disk drive unit <NUM>, a signal generation device <NUM> (e.g., a speaker), and a network interface device <NUM>.

The disk drive unit <NUM> includes a machine-readable medium <NUM> on which are stored one or more sets of data structures and instructions <NUM> (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions <NUM> can also reside, completely or at least partially, within the main memory <NUM> and/or within the processor <NUM> during execution thereof by the computer system <NUM>, with the main memory <NUM> and the processor <NUM> also constituting machine-readable media.

While the machine-readable medium <NUM> is shown in an example embodiment to be a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions <NUM> or data structures. The term "machine-readable medium" shall also be taken to include any tangible medium that is capable of storing, encoding, or carrying instructions <NUM> for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such instructions <NUM>. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media <NUM> include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

Claim 1:
A recommendation system (<NUM>) for automatically determining, within a message system in which messages are each assigned to one or more of a plurality of message channels and a plurality of users each have access to one or more of the plurality of message channels, recommendations (<NUM>) of messages to users, the system comprising:
computer memory storing vector representations (<NUM>) of the plurality of users, a plurality of authors, and the plurality of message channels, the vector representations computed based on bipartite graphs (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) each representing a relationship between a combination of the users, authors, and message channels; and
a computer processor (<NUM>) executing instructions (<NUM>) stored in the computer memory to perform operations for identifying, for each of a plurality of messages, one or more of the plurality of users to whom the message is to be recommended, the operations comprising:
performing a preliminary filtering (<NUM>), among the users of said plurality of users having access to a message channel to which the message is assigned, in order to determine a set of users, wherein the filtering is based on a user shortlist (<NUM>) associated with an author of the message, the user shortlist being based on a determination of relevance of messages from the author to users on the shortlist;
after said preliminary filtering, then scoring (<NUM>) the message for each user of the set of users with a classifier model that takes a vector representation (<NUM>) of a content of the message and the vector representations of the user, the author of the message, and the message channel associated with the message as input;
determining based on the scoring, as a subset of the set of users, the one or more of the plurality of users to whom the message is to be recommended; and
outputting the recommendation (<NUM>) to the one or more users of the plurality of users.