Recommendations using session relevance and incremental learning

In some embodiments, a computer system generates a recommendation for a user of an online service based on user actions that have been performed by the user within a threshold amount of time before the generation of the recommendation. For each user action, the computer system determines an intent classification that identifies an activity of the user and that corresponds to different types of user actions, as well as a preference classification that identifies a target of the activity, and then stores these intent and preference classifications as part of indications of the user actions for use in generating different types of recommendations using different types of recommendation models. Additionally, the computer system may use mini-batches of data from an incoming stream of logged data to train an incremental update to one or more recommendation models.

TECHNICAL FIELD

The present application relates generally to systems and methods, and computer program products for providing recommendations using real-time session relevance data and incremental learning.

BACKGROUND

Certain online services provide recommendations of digital content items to their users. In determining what digital content items to recommend to different users, some computer systems of online services use a recommendation model to select which digital content items to recommend. However, current recommendation systems are limited in their ability to provide accurate timely recommendations that are relevant to users, since they employ offline feature data pipelines to personalize recommendations, thereby resulting in delays between the occurrence of user actions that are relevant to recommendations and when the recommendation systems use such user actions as feature data in generating recommendations.

Additional technical problems arise when recommendations systems use machine learning in generating recommendations of digital content items. When observation data comes from high-velocity, user-generated data streams, machine learning methods perform a balancing act between recommendation model complexity, training time, and computational costs. Furthermore, when model freshness is critical, the training of recommendation models becomes time-constrained. Parallelized batch offline training, although horizontally scalable, is often not time considerate or computationally cost-effective. Additionally, current solutions that involve frequent offline model retraining and deployment often prove insufficient, as the computational constraints introduced to train the recommendation model hinders the ability of the underlying computer system to meet self-imposed update frequencies, and a cold-start problem (e.g., encountering entities not observed in the training data) can cause long-term repercussions for the online service, such as poor recurring recommendations influencing future engagement for new or returning users of the online service. Additionally, the performance of the recommendation model slowly degrades over time, demonstrating the need to update the recommendation model continuously.

As a result of these technical deficiencies, the functioning of the computer system of the online service is negatively affected. Other technical problems may arise as well.

DETAILED DESCRIPTION

Example methods and systems of providing recommendations using real-time session relevance data and incremental learning are disclosed. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of example embodiments. It will be evident, however, to one skilled in the art that the present embodiments may be practiced without these specific details.

Some or all of the above problems may be addressed by one or more example embodiments disclosed herein. The implementation of the features disclosed herein involves a non-generic, unconventional, and non-routine operation or combination of operations. In some example embodiments, a specially-configured recommendation system generates a recommendation for a user of an online service based on user actions that have been performed by the user within a threshold amount of time before the generation of the recommendation (e.g., within the last 24 hours before generating the recommendation). For each user action, the computer system may determine an intent classification (e.g., building a social network, finding a job) that identifies an activity of the user and that corresponds to different types of user actions, as well as a preference classification that identifies a target of the activity (e.g., another user of the online service, a type of job, a specific company), and then store these intent and preference classifications as part of indications of the user actions for use in generating recommendations. The recommendation system may use the same stored indications of user actions to generate different types of recommendations (e.g., connection recommendations, job recommendations) using different types of recommendation models (e.g., connection recommendation model, job recommendation model). By determining the corresponding intent classification and the corresponding preference classification for each user action, the recommendation system can use the same real-time session relevance data across different pages and user interface modules of an online service platform, thereby increasing the applicability of user actions in generating recommendations and improving the accuracy, timeliness, and relevance of the recommendations.

Additionally, the recommendation system may use mini-batches of data from an incoming stream of logged data to train an incremental update to one or more recommendation models. By using incremental learning updates on mini-batches of data streams, the recommendation system provides recommendation models that closely estimate a periodically-updated model trained on offline data and outperform it when recommendation model updates are time-sensitive, as well as improves the loss-function over a stale batch model. Other technical effects will be apparent from this disclosure as well.

II. Detailed Example Embodiments

The methods or embodiments disclosed herein may be implemented as a computer system having one or more modules (e.g., hardware modules or software modules). Such modules may be executed by one or more processors of the computer system. The methods or embodiments disclosed herein may be embodied as instructions stored on a machine-readable medium that, when executed by one or more processors, cause the one or more processors to perform the instructions.

FIG.1is a block diagram illustrating a client-server system100, in accordance with an example embodiment. A networked system102provides server-side functionality via a network104(e.g., the Internet or Wide Area Network (WAN)) to one or more clients.FIG.1illustrates, for example, a web client106(e.g., a browser) and a programmatic client108executing on respective client machines110and112.

An Application Program Interface (API) server114and a web server116are coupled to, and provide programmatic and web interfaces respectively to, one or more application servers118. The application servers118host one or more applications120. The application servers118are, in turn, shown to be coupled to one or more database servers124that facilitate access to one or more databases126. While the applications120are shown inFIG.1to form part of the networked system102, it will be appreciated that, in alternative embodiments, the applications120may form part of a service that is separate and distinct from the networked system102.

Further, while the system100shown inFIG.1employs a client-server architecture, the present disclosure is of course not limited to such an architecture, and could equally well find application in a distributed, or peer-to-peer, architecture system, for example. The various applications120could also be implemented as standalone software programs, which do not necessarily have networking capabilities.

The web client106accesses the various applications120via the web interface supported by the web server116. Similarly, the programmatic client108accesses the various services and functions provided by the applications120via the programmatic interface provided by the API server114.

FIG.1also illustrates a third-party application128, executing on a third-party server machine130, as having programmatic access to the networked system102via the programmatic interface provided by the API server114. For example, the third-party application128may, utilizing information retrieved from the networked system102, support one or more features or functions on a website hosted by the third-party. The third-party website may, for example, provide one or more functions that are supported by the relevant applications of the networked system102.

In some embodiments, any website referred to herein may comprise online content that may be rendered on a variety of devices, including but not limited to, a desktop personal computer, a laptop, and a mobile device (e.g., a tablet computer, smartphone, etc.). In this respect, any of these devices may be employed by a user to use the features of the present disclosure. In some embodiments, a user can use a mobile app on a mobile device (any of machines110,112, and130may be a mobile device) to access and browse online content, such as any of the online content disclosed herein. A mobile server (e.g., API server114) may communicate with the mobile app and the application server(s)118in order to make the features of the present disclosure available on the mobile device.

In some embodiments, the networked system102may comprise functional components of a social networking service.FIG.2is a block diagram showing the functional components of a social networking system210, including a data processing module referred to herein as a recommendation system216, for use in social networking system210, consistent with some embodiments of the present disclosure. In some embodiments, the recommendation system216resides on application server(s)118inFIG.1. However, it is contemplated that other configurations are also within the scope of the present disclosure.

As shown inFIG.2, a front end may comprise a user interface module (e.g., a web server)212, which receives requests from various client-computing devices, and communicates appropriate responses to the requesting client devices. For example, the user interface module(s)212may receive requests in the form of Hypertext Transfer Protocol (HTTP) requests, or other web-based, application programming interface (API) requests. In addition, a member interaction detection module213may be provided to detect various interactions that members have with different applications, services and content presented. As shown inFIG.2, upon detecting a particular interaction, the member interaction detection module213logs the interaction, including the type of interaction and any meta-data relating to the interaction, in a member activity and behavior database222.

An application logic layer may include one or more various application server modules214, which, in conjunction with the user interface module(s)212, generate various user interfaces (e.g., web pages) with data retrieved from various data sources in the data layer. With some embodiments, individual application server modules214are used to implement the functionality associated with various applications and/or services provided by the social networking service. In some example embodiments, the application logic layer includes the recommendation system216.

As shown inFIG.2, a data layer may include several databases, such as a database218for storing profile data, including both member profile data and profile data for various organizations (e.g., companies, schools, etc.). Consistent with some embodiments, when a person initially registers to become a member of the social networking service, the person will be prompted to provide some personal information, such as his or her name, age (e.g., birthdate), gender, interests, contact information, home town, address, the names of the member's spouse and/or family members, educational background (e.g., schools, majors, matriculation and/or graduation dates, etc.), employment history, skills, professional organizations, and so on. This information is stored, for example, in the database218. Similarly, when a representative of an organization initially registers the organization with the social networking service, the representative may be prompted to provide certain information about the organization. This information may be stored, for example, in the database218, or another database (not shown). In some example embodiments, the profile data may be processed (e.g., in the background or offline) to generate various derived profile data. For example, if a member has provided information about various job titles the member has held with the same company or different companies, and for how long, this information can be used to infer or derive a member profile attribute indicating the member's overall seniority level, or seniority level within a particular company. In some example embodiments, importing or otherwise accessing data from one or more externally hosted data sources may enhance profile data for both members and organizations. For instance, with companies in particular, financial data may be imported from one or more external data sources and made part of a company's profile.

Once registered, a member may invite other members, or be invited by other members, to connect via the social networking service. A “connection” may require or indicate a bi-lateral agreement by the members, such that both members acknowledge the establishment of the connection. Similarly, with some embodiments, a member may elect to “follow” another member. In contrast to establishing a connection, the concept of “following” another member typically is a unilateral operation, and at least with some embodiments, does not require acknowledgement or approval by the member that is being followed. When one member follows another, the member who is following may receive status updates (e.g., in an activity or content stream) or other messages published by the member being followed or relating to various activities undertaken by the member being followed. Similarly, when a member follows an organization, the member becomes eligible to receive messages or status updates published on behalf of the organization. For instance, messages or status updates published on behalf of an organization that a member is following will appear in the member's personalized data feed, commonly referred to as an activity stream or content stream. In any case, the various associations and relationships that the members establish with other members, or with other entities and objects, are stored and maintained within a social graph, shown inFIG.2with database220.

As members interact with the various applications, services, and content made available via the social networking system210, the members' interactions and behavior (e.g., content viewed, links or buttons selected, messages responded to, etc.) may be tracked and information concerning the member's activities and behavior may be logged or stored, for example, as indicated inFIG.2by the database222. This logged activity information may then be used by the recommendation system216. The members' interactions and behavior may also be tracked, stored, and used by the recommendation system216residing on a client device, such as within a browser of the client device, as will be discussed in further detail below.

In some embodiments, databases218,220, and222may be incorporated into database(s)126inFIG.1. However, other configurations are also within the scope of the present disclosure.

Although not shown, in some embodiments, the social networking system210provides an application programming interface (API) module via which applications and services can access various data and services provided or maintained by the social networking service. For example, using an API, an application may be able to request and/or receive one or more navigation recommendations. Such applications may be browser-based applications or may be operating system-specific. In particular, some applications may reside and execute (at least partially) on one or more mobile devices (e.g., phone, or tablet computing devices) with a mobile operating system. Furthermore, while in many cases the applications or services that leverage the API may be applications and services that are developed and maintained by the entity operating the social networking service, other than data privacy concerns, nothing prevents the API from being provided to the public or to certain third-parties under special arrangements, thereby making the navigation recommendations available to third-party applications and services.

Although the recommendation system216is referred to herein as being used in the context of a social networking service, it is contemplated that it may also be employed in the context of any website or online services. Additionally, although features of the present disclosure can be used or presented in the context of a web page, it is contemplated that any user interface view (e.g., a user interface on a mobile device or on desktop software) is within the scope of the present disclosure.

In some embodiments, the recommendation system216resides on a computer system, or other machine, having a memory and at least one processor (not shown). The recommendation system216may be incorporated into the networked system102, such as in the application server(s)118inFIG.1. In some example embodiments, the recommendation system216includes, or is otherwise in communication with, any combination of one or more of databases218,220, and222inFIG.2. Other configurations of the recommendation system216are also within the scope of the present disclosure.

In some example embodiments, the recommendation system216is configured to provide a variety of user interface functionality, such as generating user interfaces, interactively presenting user interfaces to the user, receiving information from the user (e.g., interactions with user interfaces), and so on. Presenting information to the user can include causing presentation of information to the user (e.g., communicating information to a device with instructions to present the information to the user). Information may be presented using a variety of means including visually displaying information and using other device outputs (e.g., audio, tactile, and so forth). Similarly, information may be received via a variety of means including alphanumeric input or other device input (e.g., one or more touch screen, camera, tactile sensors, light sensors, infrared sensors, biometric sensors, microphone, gyroscope, accelerometer, other sensors, and so forth). In some example embodiments, the recommendation system216is configured to receive user input. For example, the recommendation system216can present one or more GUI elements (e.g., drop-down menu, selectable buttons, text field) with which a user can submit input.

In some example embodiments, the recommendation system216is configured to perform various communication functions to facilitate the functionality described herein, such as by communicating with the social networking system210via the network104using a wired or wireless connection. The recommendation system216may also provide various web services or functions, such as retrieving information from the third party servers130and the social networking system210. Information retrieved by the recommendation system216may include profile data corresponding to users and members of the social networking service of the social networking system210.

Additionally, the recommendation system216can provide various data functionality, such as exchanging information with databases or servers. For example, the recommendation system216can access member profiles that include profile data from databases, such as from the database218, as well as extract attributes and/or characteristics from the profile data of member profiles. Furthermore, the recommendation system216can access social graph data and member activity and behavior data from databases, such as the databases220and222, as well as exchange information with third party servers130, client machines110,112, and other sources of information.

FIG.3is a block diagram illustrating components of a recommendation system216, in accordance with an example embodiment. In some embodiments, the recommendation system216comprises any combination of one or more of a selection component310, a frontend component320, an activity extractor330, a real-time store340, an activity processor350, an other feature store360, and a coefficient store370. The recommendation system216can reside on a computer system, or other machine, having a memory and at least one processor (not shown). In some embodiments, the recommendation system216can be incorporated into the application server(s)118inFIG.1. However, it is contemplated that other configurations of the recommendation system216are also within the scope of the present disclosure.

In some example embodiments, the recommendation system216is configured to provide a variety of user interface functionality, such as generating user interfaces, interactively presenting user interfaces to the user, receiving information from the user (e.g., interactions with user interfaces), and so on. Presenting information to the user can include causing presentation of information to the user (e.g., communicating information to a device with instructions to present the information to the user). Information may be presented using a variety of means including visually displaying information and using other device outputs (e.g., audio, tactile, and so forth). Similarly, information may be received via a variety of means including alphanumeric input or other device input (e.g., one or more touch screen, camera, tactile sensors, light sensors, infrared sensors, biometric sensors, microphone, gyroscope, accelerometer, other sensors, and so forth). In some example embodiments, the recommendation system216is configured to receive user input. For example, the recommendation system216can present one or more GUI elements (e.g., drop-down menu, selectable buttons, text field) with which a user can submit input.

In some example embodiments, the recommendation system216is configured to perform various communication functions to facilitate the functionality described herein, such as by communicating with the social networking system210via the network104using a wired or wireless connection. The recommendation system216may also provide various web services or functions, such as retrieving information from the third party servers130and the social networking system210. Information retrieved by the recommendation system216may include profile data, social graph data, and member activity and behaviour data corresponding to users and members of the social networking service of the social networking system210.

In some example embodiments, the selection component310is configured to generation one or more recommendations for a user of an online service. The selection component310may cause the generated recommendation(s) to be displayed on a computing device305of the user. In some example embodiments, the selection component310may use different recommendation models (e.g., MODEL-1, . . . , MODEL-N) to generate different types of recommendations, such as a connection recommendation model to generate a connection recommendation for the user and a job recommendation model to generate job recommendations for the user.

FIG.4illustrates a graphical user interface (GUI)500in which a personalized feed of a user is displayed on a home page of an online service, in accordance with an example embodiment. The personalized feed of the user comprises digital content items410of different types. For example, inFIG.5, the personalized feed comprises a first digital content item410A of a first type, a second digital content item410B of a second type different from the first type, and a third digital content item410C of a third type different from both the first type and the second type. The digital content items410displayed as part of the personalized feed may be generated by the recommendation system216. In some example embodiments, the selection component310uses a feed recommendation model to generate the recommendations to be displayed in the personalized feed of the user. Although the feed recommendation model may leverage output from other recommendation models in generating recommendations to be displayed in the personalized feed of the user, such as output from a connection recommendation model and output from a job recommendation model, the feed recommendation model may still be considered its own discrete model.

FIG.5illustrates a GUI500in which recommendations510of connections with other users selected using the recommendation system216are displayed on a connection recommendation page of an online service, in accordance with an example embodiment. InFIG.5, each recommendation510may comprise an identification of the corresponding recommended other user, an image associated with a profile of the corresponding recommended other user, one or more attributes of the corresponding recommended other user (e.g., job position, company), and a selectable user interface element (e.g., a clickable “CONNECT” button) configured to cause a user-to-user message (e.g., an invitation to connect) to be transmitted to the corresponding recommended other user or to cause another type of user action to be performed. Each recommendation510may also comprise a selectable user interface element (not shown) configured to reject or otherwise dismiss the corresponding recommendation, so as to indicate an instruction by the user not to perform the user action for the corresponding recommended other user of the corresponding recommendation. The recommendations510of connections may be generated using a type of recommendation model dedicated to generating only recommendations510of connections.

FIG.6illustrates a GUI600in which recommendations610of online job postings selected using the recommendation system216are displayed on a job recommendation page of an online service, in accordance with an example embodiment. In some example embodiments, the online job postings are published on the online service, and the recommendations610are caused to be displayed to the user on the computing device of the user. The recommendations610may each comprise one or more corresponding selectable user interface elements (e.g., hyperlinked text) configured to display more information about the corresponding online job posting of the recommendation610(e.g., to view the entire online job posting rather than just an abbreviated summary of the online job posting) or to enable the user to perform some other type of online action directed towards the online job posting of the recommendation610, such as saving the online job posting or applying to the online job posting. Each recommendation610may include information about the corresponding online job posting, such as a job title, a company name, a geographical location, and desired skills, educational background, and work experience. Other types of information may also be included in the recommendation610. The GUI600may also display one or more user interface elements620configured to enable the user to submit a search query for searching for online job postings, such as by entering keyword search terms into a search field. In response to one or more keywords being submitted by the user as part of a search query via the search field, the recommendation system216may generate recommendations610based on the keyword(s) and feature data of online job postings being evaluated as search results. The recommendations610of online job postings may be generated using a type of recommendation model dedicated to generating only recommendations610of online job postings.

In some example embodiments, the recommendation system216is configured to determine the short-term intent and preferences of the user based on real-time processing of actions of the user, where the intent of the user comprises an activity of the user and corresponds to different types of user actions, and the preferences of the user comprise one or more targets of the activity. When the user takes an action on the computing device305, the computing device305makes a call to the frontend component320. Examples of user actions include, but are not limited to, posting a comment for a digital content item posted by another user, sending an invitation to connect to another user, following another user, submitting an electronic like of a digital content item posted by another user, sharing a digital content item posted by another user, sending a message to another user, viewing an online job posting via the online service, and applying to an online job posting via the online service. Other types of user actions are also within the scope of the present disclosure.

The frontend component305determines if the user action is worth recording in the real-time store340. For example, certain actions may be determined to not be indicative of the user's short-term interests or preferences. In some example embodiments, the frontend component320uses at least one of a whitelist of user actions to determine which user actions to record in the real-time store340for use as real-time relevance data in generating recommendations. If the frontend component320determines that a user action is worth recording, such as based on a determination that the user action is on the whitelist, then the frontend component320makes a to call to the activity extractor330, which extracts the raw action data from an event object triggered by the user action. The real-time store340may comprise a real-time activities store342configured to store raw data and a real-time features store344configured to store feature data corresponding to the raw data, such as a vector representation based on the raw data. The raw action data is stored in the real-time activities store342with a fixed time-to-live. Upon receiving the raw action from the activity extractor330, the activity processor350reads the latest real-time feature data of the user from the real-time features store and modifies it based on the new user action. The new feature data are written back to the real-time features store344along with their last modified time and time-to-live.

In some example embodiments, the raw data stored in the real-time activities store342and the feature data stored in the real-time features store344comprise corresponding indications of the user action. Each indication of a user action may comprise an intent classification and a preference classification for the corresponding user action. The intent classification identifies an activity of the user. In some example embodiments, the storing of the corresponding indication of the user action comprises selecting the corresponding intent classification from a plurality of intent classifications based on the corresponding user action. The plurality of intent classifications may comprise building a social network, finding a job, staying informed, and nurturing a relationship. However, the plurality of intent classifications may comprise other configurations of intent classifications. In some example embodiments, each intent classification corresponds to a plurality of different types of user actions. For example, one intent classification may be finding a job, which corresponds to the user actions of viewing an online job posting, saving an online job posting, and applying to an online job posting.

In some example embodiments, the preference classification identifies a target of the activity. The storing of the corresponding indication of the user action may comprise selecting the corresponding preference classification from a plurality of preference classifications based on the corresponding user action. The plurality of preference classifications may comprise another user of the online service, a type of job, a specific company, a type of company, an industry, and a geographic location. However, the plurality of preference classifications may comprise other configurations of preference classifications.

The recommendation system216may use the user actions to build a short-term preference map and a short-term intent map for the user.FIG.7is a table700illustrating user actions710building a short-term preference map720and a short-term intent map730for use in a recommendation system, in accordance with an example embodiment. InFIG.7, the table shows a user action710of the user starting a session by clicking on an invitation to connect from a first other user (e.g., MEMBER-1) resulting in an indication in the short-term preference map720of a short-term preference for the first other user, as well as an indication in the short-term intent map730for building a network (e.g., BUILDING_NETWORK). Next, a user action710of the user applying to an online job posting at a company (e.g., COMPANY-1) results in an indication in the short-term preference map720of a short-term preference for the company, as well as an indication in the short-term intent map730for finding a job (e.g., FINDING JOB). Then, a user action710of the user following the company results in another indication in the short-term preference map720of a short-term preference for the company (e.g., COMPANY-1), as well as another indication in the short-term intent map730for building a network (e.g., BUILDING_NETWORK). Next, a user action710of the user liking a feed update from a second other user (e.g., MEMBER-2) results in an indication in the short-term preference map720of a short-term preference for the second other user, as well as an indication in the short-term intent map730for staying informed (e.g., STAYING_INFORMED).

InFIG.7, the short-term preference map720and the short-term intent map730show a raw count being maintained for each respective preference classification and intent classification, where the raw count follows the colon (“:”). In some example embodiments, the recommendation system216processes these raw counts to generate vector representations of the short-term preference map720and the short-term intent map730, which may be stored in the real-time features store344as real-time feature vectors. The selection component310may use these real-time feature vectors from the real-time features store344or the raw action data from the real-time activities store342, along with one or more other feature data from the other features store360(e.g., profile data of the user, social graph data of the user, information about subject of digital content item to be recommended) to generate a recommendation for the user based on a recommendation model. In some example embodiments, the selection component310selects, from amongst a plurality of recommendations models of the online service, the recommendation model to use in generating the recommendation for the user. For example, the plurality of recommendation models may comprise a model configured to be used to generate recommendations of digital content items to be displayed within a personalized feed of the user on the home page, another model configured to be used to generate recommendations of other users with whom to form connections, and yet another model configured to be used to generate recommendations of online job postings. In some example embodiments, the recommendation system216selects the recommendation model based on an identification of a particular page or user interface module (e.g., a particular bounded area within a page dedicated to display of a particular type of recommendation) in which the recommendation is to be provided.

In some example embodiments, the recommendation system216records raw actions of users as (actor, verb, object) tuples. For example, (member:2, invited, member:3). A record of an action may be available for processing and use in generating recommendations for a user within 30 seconds after the action is performed by the user. The recommendation system216may select the objects in the action tuples to include in the short-term preference map730, and may select the verbs in the action tuples to include in the short-term intent map720.

The selection component310uses the stored indications of the user actions as feature data for the recommendation model based on the user actions having been performed within a threshold amount of time before the point in time at which the recommendation is generated. For example, the recommendation system216may filter out any indications of user actions not performed within the last 24 hours, and then use only the indication user actions performed within the last 24 hours as feature data for the recommendation model in generating the recommendation.

In some example embodiments, the selection component310causes the generated recommendation to be displayed on the computing device305. The selection component310may cause recommendations of different recommendation models to be displayed on different pages of the online service, even though the same indications of user actions are used to generate the different recommendations. For example, one or more recommendations may be generated based on a feed recommendation model using the stored indications of user actions and displayed on a home page of the online service that is displayed on the computing device305of in response to the user logging onto the online service (e.g. on the home page shown inFIG.4), and one or more other recommendations may generated by another type of recommendation model (e.g., a connection recommendation model, a job recommendation model) and displayed on any other page of the online service that is navigated to from the home page via selection by the user of at least one selectable link (e.g., on the connection recommendation page shown inFIG.5, on the job recommendation page shown inFIG.6).

In generating recommendations for a user, the selection component310may prioritize recommendations of digital content items from the entities that are present in the short-term preference map720for the user, as well as prioritize recommendations of digital content items that satisfy the user's short-term intent based on the short-term intent map730. In some example embodiments, the recommendation system216may analyze the user action to infer additional preferences of the user based on similarities or connections between the target of the user action and additional entities. In one example, the user starts a session with the online service by clicking an invitation to connect from another user labeled MEMBER-1. MEMBER-1:1 is added to the short-term preference map720. Now, for every user (MEMBER-J) in the user's social network (e.g., every other user connected to the user in the user's social graph), the recommendation system216may compute a similarity value s (e.g., such as a percentage) between the feature vector of MEMBER-1 and the feature vector of MEMBER-J, and then add MEMBER-J: s to the short-term preference map.

In some example embodiments, the selection component310uses one or more coefficients stored in the coefficient store370in generating the recommendation. The coefficients are used by the selection component310as weights for the feature data used by the recommendation model in generating the recommendation. In generating recommendations, the selection component310may use Generalized Additive Mixed Effect (GAME) models, which comprise a large fixed-effects model that is trained on a whole dataset to learn the probability of engagement for a particular viewer and content, as well as a series of simpler linear random-effects models trained on data corresponding to each entity (e.g., per-viewer or per-content) for more granular personalization. The fixed-effects part of the model can be arbitrarily complex (e.g., wide & deep, boosted trees, ensembles, or linear) depending on the targeted task. The random-effect models are a powerful way to personalize both member and content patterns. However, in cases where models are fit to incredibly large datasets, the additional model-complexity comes with large computational costs.

Furthermore, to model newly introduced content or content with short-lived significance, computation time becomes a valuable commodity, and limiting it causes a computationally-constrained learning problem. For example, a new advertising campaign can reach thousands of impressions in a few minutes. Similarly, a job posting can reach millions of job seekers in a short time window. In these cases, recommendation models need to learn from the information as quickly as possible. Specifically, a cold start problem and a data shift problem have been observed when estimating viewer engagement. In an example of the cold start problem, over the course of a day, users of the online service encounter advertisers and advertisements that are not observed in the training set. These digital content items can be brand-new advertisers and advertisements, or advertisers who have not marketed through the online service for extended periods of time. Under these scenarios, random-effect models must quickly adapt to new data. In an example of the data shift problem, viewers' most recent engagements are the best interest indicators, as interest can dramatically shift over short periods of time. For example, a viewer who might be heavily engaging with advertisements about cloud services on the online service may be less inclined to engage immediately after signing up with some of the cloud services.

Solutions such as frequent offline model retraining and deployment often proves insufficient for the following reasons. First, the computational constraints introduced to train the model disallows the recommendation system216from meeting self-imposed update frequencies. Second, the cold-start problem can cause long-term repercussions for the online service, such as poor recurring recommendations influencing future engagement for new or returning dormant members. Additionally, the model performance slowly degrades over time, demonstrating the need to update the model continuously. In some example embodiments, the recommendation system216implements a framework to incrementally train a recommendation model, incrementally updating models between full batch offline updates of the recommendation model to best balance performance against stability.

In some example embodiments, the recommendation system216prepares mini-batches of data from an incoming stream of logged data to train an incremental update to the coefficients of the recommendation model stored in the coefficient store370. Since technical problems arise in holding the entire dataset (e.g., all historical data) in memory, the recommendation system216may approximate the loss function on previously observed data by a local quadratic approximation around the previous optimal and combine it with the loss function on the most recent mini-batch. This approach results in a regularized learning problem with a weighted penalty. In the Bayesian interpretation, the local quadratic approximation is equivalent to using the posterior distribution of the previous model as prior for training the update.

The present disclosure uses the following notation We refer to a training dataset Ω(t)as the mini-batch/slice of ordered “data instances” observed in time interval t. Each Ω(t)consists of data instances represented by the tuple (xn,yn), where xnis an observed feature vector describing instance n, and ynis a corresponding observed label. We denote the cumulative training dataset observed until time t usingΩ(t)=Ω(t-1)∪Ω(t). We use β*(t)to refer to a parameterized model trained on the full datasetΩ(t), and βinc(t)to refer to the incremental model trained using the incremental learning framework of the present disclosure.

In the GAME framework, the model β=[βƒ, B] consists of two components: a fixed-effects model, βƒ, and a set of random-effect models, B={Br}r∈R, where R denotes the set of random-effect types. Coefficients of a single random-effect type r are denoted by Br={Brl}l=1Nr, where Nris the total number of random-effects with random-effect type, r (e.g., r may correspond to per-viewer, per-content, or per-job; and l is a specific viewer, content or job, and Nris the number of viewers, content, or jobs). Lastly, for each data instance xn,yn, we use i(r,n) to represent the index value of random-effect type r for instance n.

Given a stream of observation data Ω(0), Ω(1), . . . , Ω(t), . . . , and a base convex loss function ƒ (e.g., log-logistic), we define a series of loss functions over the mini-batch of datastream Ω(t)as

ft⁡(β)=∑n∈Ω(t)⁢f⁡(xn,yn,β),
and a time-discounted cumulative loss over the total dataΩ(t)

Ft⁡(β)=∑k=0k=t⁢δt-k⁢fk⁡(β)=δ⁢Ft-1⁡(β)+ft⁡(β),
where δ<1 is a time discount or forgetting factor to emphasize the fresher data.

Generalized Linear Mixed (GLMix) models may be used as an approach for personalizing traditional Generalized Linear Models (GLM's) through the addition of linear random-effects models keyed to discretely valued random effect variables. Example random-effects variables range from unique entity identifiers (e.g., advertisement, job, member) to contextual information, such as the consumption device (e.g., desktop, mobile application, etc.). In practice, the coefficients may fit conditionally on the presence of these variables and serve a supplementary role in providing personalization atop the GLM.

In some example embodiments, the recommendation system216solves for the fixed-effect coefficients, βƒ, and the random-effect coefficients, B, through the optimization of

minimizeβ=[βf,B]⁢∑n∈Ω_⁢-log⁢p⁡(yn|sn)-log⁢p⁡(βf)-∑r∈R⁢∑l=1Nr⁢log⁢p⁡(βr⁢l)
where snis defined as

sn=xnT⁢βf+∑r∈R⁢zmT⁢βr,i⁡(r,n),
with xnrepresenting the feature vector for the fixed-effect on sample n, zrnrepresenting the feature vector for random-effects type r on n, and βr,i(r,n), corresponding to the coefficients of the random-effects value i(r,n). The probability terms log p(βƒ) and log p(βrl) results from a standard Gaussian prior N(0,λI) on all the coefficients.

Although the GLMix formulations in the preceding two equations above describe fixed-effects and random-effects coefficients as linear models, they are not limited to such. By substituting snfor:

Incremental learning describes the general set of algorithms attempting to solve large-scale machine learning through immediate model updates in response to new data instances without additional passes over pre-existing data. These scenarios occur when training models on large streams of data, where there is a limited time window for models to adjust, constraining computational cost of the learning process. Model update rates can be controlled over batch sizes of new observations (e.g., incremental batch learning) with a batch size of 1 corresponding to online learning.

In some example embodiments, the recommendation system216provides an efficient solution for updating the random-effects model as fresher data flows. One aspect is the parallelization of random-effects model training, which is possible when data instances are grouped by random-effects values.

For every batch of data Ω(t), the recommendation system216may solve the problem

minimizeβ⁢⁢Ft⁡(β)-log⁢p⁡(β),
where p(β) is the prior distribution of the model parameters β. Assuming a Gaussian prior p(β)˜N(0,λI), in the case of log-logistic loss, the output of the problem is a Gaussian posterior distribution pt(β)˜N(β*(t), Σ*(t)) (using Laplace approximation).

However, as the data size grows, it is increasingly difficult to hold it all in memory and computationally intensive to iterate over the entire data set for computing full Ft, ∇Ft, and/or ∇2Ft. Therefore, the recommendation system216may maintain an approximation of the loss function over the past observations, Ft-1, and combine it with the loss function over the latest batch ƒt. In particular, the recommendation system216may split the problem into the loss on the historical data and the loss on incremental batch,

minimizeβ⁢⁢δ⁢⁢Ft-1⁡(β)+ft⁡(β)-log⁢p⁡(β),
The first term of objective function can be approximated using a quadratic function centered around previous optimum β*(t-1),

minimizeβ⁢⁢δ2⁢β-β*(t-1)∇2⁢Ft-1⁡(β⁢*(t-1))2+ft⁡(β)+λ2⁢β2.
The optimality of β*(t-1)implies ∇Ft-1(β*(t-1))=0 (hence, the linear term is dropped). In the Bayesian framework, since (Σ*(t-1))−1=∇2Ft-1(β*(t-1)), this approximate problem is equivalent to training the mini-batch using the posterior distribution, N(β*(t-1), Σ*(t-1)), from the previous step as a prior distribution for β(t).

Additionally, in the GAME framework, the recommendation system216may use asynchronous updates for the random-effect models to keep them fresh in production systems. The recommendation system216may control the rate at which a random-effects model is updated by a mini batch-size parameter. Once a sufficiently-sized batch of data Ωrl(t)={n∈Ω(t)|i(r,n)=l}, for random-effectrand indexl, has been observed, the recommendation system216updates the corresponding random-effect by solving:

FIG.8is a block diagram illustrating components that implement incremental learning in the recommendation system216, in accordance with an example embodiment. In some example embodiments, the recommendation system216comprises a feature tracking component810, a nearline training data generation component820, a fixed-effects model scoring component830, and a random-effects model training component840.

Although the fixed-effects and random-effects models may be retrained offline as often as possible, only restricted by the time cost to complete one full training session, in practice, it has been observed to take anywhere between 8-24 hours depending on the size of the data and the complexity of the recommendation model. The frequency of updating the random-effect coefficients B (e.g., the coefficients stored in the coefficient store370) is restricted by the time required to process the data and train the random-effects models, which is largely proportional to the size of the training data, which is itself a function of the time intervals, T. The recommendation system216may use the incremental learning framework of the present disclosure to reduce the time required to adapt to the nearline observations by processing incremental data and learning on sequences of micro-batches.

In some model training pipelines, labeled data instances (“observations”) are joined with offline stores of uniquely keyed features (e.g., member features, ad features, etc.) to produce the training data, Ω. Models trained through this method are applied on-demand through an online serving layer, where feature producer functions produce a real-time representation of xnprior to being scored with the model. However, there are technical problems with this approach, such as differences between offline/online feature productions (e.g., due to bugs, versioning, unavailability, etc.), as well as the temporal aspect of certain feature sets. Training pipelines that rely on the present value of time sensitive features (e.g., activity-based features) can cause severe overfitting, and, in the worst-cases, introduce label information into training data.

As a technical solution, in some example embodiments, the recommendation system216employs an online feature tracking system in the form of the feature tracking component810that tracks the feature values when the response prediction service scores a digital content item (e.g., an advertisement) for recommendation using an open-source stream-processing software platform that is used to serve real-time data for online services. Feature tracking guarantees feature parity since feature values are acquired in a single fetch. For online services that already score and fetch features on-demand, the amount of additional work to log feature values is very low.

In some example embodiments, an open-source stream-processing software platform of the online service provides topics such as impressions, clicks, and tracked raw data. Machine learning applications may use aggregates or derived statistics from this raw data. The nearline training data generation component820listens to multiple topics and produces featurized training examples. Next, the nearline training data generation component820publishes to a new topic, which is processed through an extract, transform, load (ETL) layer and dumped to storage for offline consumption. As part of the processing logic, the job is the single source of truth for label cleaning, attribution as well as stream-table joins to enrich the data using external datasets. Features that are fairly static, but dense (e.g., embeddings of static entities) are good candidates to be introduced at this stage since it is cost-inefficient to log highly redundant, dense features.

Two knobs that may be fine-tuned by analyzing traffic patterns for data quality are: (1) trade-off between training cost and accuracy, and (2) scale for high throughput. With respect to the trade-off between training cost and accuracy, since offline data pipelines have access to “delayed” clicks while preparing the training data, all the clicks are accounted for. In the streaming case, the use of windowing limits the clicks that would be considered for training. The longer the window, the higher the cost and lower the rate of missed data. With respect to scale for high throughput, the system has to process the input events fast enough so that the storage used for queuing the events will not exceed capacity. The data processing is designed to minimize data shuffle across the network to ensure scalability. This solution also ensures that all the data corresponding to an advertiser is processed on the same node.

In some example embodiments, the fixed-effect model scoring component830comprises a stream processor that listens on a stream of training data and scores each incoming instance with the fixed-effects model parameterized by βƒ(0). If the fixed-effects model is an ensemble, the entire ensemble is scored just like it would be online to compute a score for every impression. The score serves as the offset for random-effects training, as will be explained in further detail below with respect to Algorithm 1 ofFIG.8. The stream processor of the fixed-effect model scoring component830, at this stage, batches the incoming data keyed by the random-effect identification (e.g., advertiser) and with configurable triggers (e.g., based on impressions, time since last update or coefficient variance) to kick start an incremental model training.

The random-effect model training component840performs incremental model training using an incremental learning algorithm.FIG.9illustrates an incremental learning algorithm (Algorithm 1) for the recommendation system216, in accordance with an example embodiment. In lines2-7of the incremental learning algorithm inFIG.9, the random-effect model training component840initializes the parameters and train the fixed and random effect models with batch data. In production systems, this initial set of training may be performed in an offline system and models may be deployed to online services. In lines9-10of the incremental learning algorithm inFIG.9, the random-effect model training component840gets the next mini-batch data and computes the offset coming from the fixed-effect model of the fixed-effect model scoring component830. The recommendation system216may maintain the same stationary βƒ(0)for all t. The offset provides the connection of the fixed-effects model with the random-effects model (e.g., the coefficients of the random-effects model) that are to be updated. This approach enables the recommendation system216to use any other parametric machine learning model for fixed-effect representation, such as a deep neural network or a boosted decision trees. In lines11-16of the incremental learning algorithm inFIG.9, the random-effect model training component840retrains the incremental models to update the posterior distribution. For online serving, the random-effect model training component840may use βƒ(0)and Thompson Sampling from the updated distribution for βr(t). For each mini-batch of data Ω(t), the random-effect model training component840solves the inner loop of Algorithm 1 inFIG.9to update β*r(t)and Σ*r(t). The learned coefficients may be written to a read-write key-value store, such as the coefficient store370, from which the selection component310may read at recommendation request time.

The recommendation system216may provide an advertising platform in which customers can target hundreds of millions of members of the online service via digital content items comprising sponsored updates. The recommendation models of the recommendation system216play a central role to score these digital content items by estimating the engagement rate of members. The production system for the recommendation models can be logically separated into two phases: (1) the cold start phase where the fixed-effects part of the GAME model is trained, and (2) the warm start phase trains random-effects coefficients per ad to boost the scores of the fixed-effects model. The fixed-effects model training identifies a member's engagement, propensity to click, and interactivity within the online service's ecosystem through attributes of their profile, historical behavior, and contextual cues. These fixed-effects models are typically large non-linear models periodically retrained using logged historical data. The warm start models capture information specific to the ad, campaign, and advertiser. These random-effects models provide fine-grained modeling control at different levels of granularity.

Since the random-effects training is for every individual advertiser, it can be trained incrementally to only account for the delta (e.g., newly added advertisement content) and avoid redundant training. Ideally, the coefficients should be retrained as frequently as possible because the random effects of the GAME models are sensitive to the retrain frequency, especially for new advertisers or creatives that have not been seen previously. However, since the incremental loss formulation is an approximation, the error can propagate and make the model worse. Thus, it is important to find the sweet spot where the recommendation system216benefits from updating the recommendation model more frequently, but does not go through too many slices of incremental updates before the offline model refreshes. The recommendation system216may use offline batch model training for Ω(0)and online model serving.

Offline batch model training for Ω(0)involves training an initial GAME model, β*(0)=[β*ƒ(0),{β*r(0)}r∈R]. In one embodiment, the recommendation system216trains a GAME model Ω(0)over a training window of T days (e.g., 14-30 days, depending on naturally occurring periodicity). Since βƒ(0)represents a generalized fixed-effect model learned across a complete dataset, it is tolerant to minor data shifts and can be updated less frequently (e.g., once every d days). However, new items get added to the system constantly and item-specific behaviors are fast changing. Therefore, {Br}r∈Rhas to be updated frequently, every h hours (e.g., 6-8 hours) to capture these changes.

In some example embodiments, for each training cadence of d days, the recommendation system216starts fresh and train β*(0)[βƒ(0),{β*r(0)}r∈R] on the observations data Ω(0)collected over training window T by solving the following equation:

minimizeβ=[βf,B]⁢∑n∈Ω_⁢-log⁢p⁡(yn|sn)-log⁢p⁡(βf)-∑r∈R⁢∑l=1Nr⁢log⁢p⁡(βr⁢l).
Turning to the online model serving component, the training data has varying amount of samples for different random-effects resulting in different covariance estimates for βr's. The recommendation system216may use a Thompson Sampling scheme to incorporate these covariance estimates in online inference. For each of the random-effects coefficients, the recommendation system216may use the posterior mean β*r(0), and the variance Σ*r(0)to sample a βrfor online inference and serving.

FIG.10is a flowchart illustrating a method of providing recommendations, in accordance with an example embodiment. The method1000can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device), or a combination thereof. In one implementation, the method1000is performed by the recommendation system216ofFIGS.2,3, and8, as described above.

At operation1010, the recommendation system216, for each one of a plurality of user actions performed on an online service by a user via a computing device of the user, stores a corresponding indication of the user action in response to the user action being performed. In some example embodiments, the corresponding indication for each user action comprises an intent classification and a preference classification for the corresponding user action. The intent classification identifies an activity of the user and corresponds to different types of user actions, whereas the preference classification identifies a target of the activity. The intent classification may comprise an activity that the user performed or attempted to perform via one or more types of user actions, thus representing an intended activity of the user. For example, one intent classification may be finding a job, which may correspond to the user actions of viewing an online job posting, saving an online job posting, and applying to an online job posting. In some example embodiments, the plurality of user actions comprise at least one of posting a comment for a digital content item posted by another user, sending an invitation to connect to another user, following another user, submitting an electronic like of a digital content item posted by another user, sharing a digital content item posted by another user, sending a message to another user, viewing an online job posting via the online service, and applying to an online job posting via the online service. However, other types of user actions are also within the scope of the present disclosure.

In some example embodiments, the storing of the corresponding indication of the user action comprises selecting the corresponding intent classification from a plurality of intent classifications based on the corresponding user action. The plurality of intent classifications may comprise building a social network, finding a job, staying informed, and nurturing a relationship. However, the plurality of intent classifications may comprise other configurations of intent classifications. In some example embodiments, the storing of the corresponding indication of the user action comprises selecting the corresponding preference classification from a plurality of preference classifications based on the corresponding user action. The plurality of preference classifications may comprise another user of the online service, a type of job, a specific company, a type of company, an industry, and a geographic location. However, the plurality of preference classifications may comprise other configurations of preference classifications.

At operation1020, the recommendation system216selects, from amongst a plurality of recommendations models of the online service, a first recommendation model to use in generating a recommendation for the user. For example, the plurality of recommendation models may comprise a model configured to be used to generate recommendations of digital content items to be displayed within a personalized feed of the user on the home page, another model configured to be used to generate recommendations of other users with whom to form connections, and yet another model configured to be used to generate recommendations of online job postings. In some example embodiments, the recommendation system216selects the recommendation model based on an identification of a particular page or user interface module (e.g., a particular bounded area within a page dedicated to display of a particular type of recommendation) in which the recommendation is to be provided.

At operation1030, the recommendation system216generates, at a first point in time, a recommendation of a first type based on the selected first recommendation model. In some example embodiments, the generating of the first recommendation comprises using the stored indications of the plurality of user actions as feature data for the first recommendation model based on the plurality of user actions having been performed within a threshold amount of time before the first point in time. For example, the recommendation system216may filter out any indications of user actions not performed within the last 24 hours, and then use only the indication user actions performed within the last 24 hours as feature data for the first recommendation model in generating the recommendation.

At operation1040, the recommendation system216causes the first recommendation to be displayed on the computing device. In some example embodiments, the causing the first recommendation to be displayed on the computing device comprises causing the first recommendation to be displayed on a first page of the online service. For example, the first recommendation may be displayed on a home page of the online service displayed on the computing device of the user in response to the user logging onto the online service (e.g. on the home page shown inFIG.4) or on any page that is navigated to from the home page via selection by the user of at least one selectable link, such as a connection recommendation page (e.g., on the connection recommendation page shown inFIG.5) or a job recommendation page (e.g., on the job recommendation page shown inFIG.6).

The method1000may return to operation1010to store one or more additional indications of user actions performed by the user via the online service since the previous performance of operation1010. The method1000may, again, select a recommendation model at operation1020, such as by selecting a different recommendation model than previously selected in the preceding performance of operation1020, and then, at operation1030, generate a recommendation using the newly-selected recommendation model and the most recent version of stored indications of user actions. This subsequent recommendation may be a recommendation of a different type than the recommendation of the previous performance of operation1030. In one example, a connection recommendation is generated during the previous performance of operation1030in the preceding cycle of the method1000(e.g., at a first point in time), and a job recommendation is generated during the current performance of operation1030in the current cycle of the method1000(e.g., at a second point in time after the first point in time, such as 10 minutes after the first point in time). The recommendation system216then causes the newly-generated recommendation to be displayed on the computing device of the user.

At operation1050, the recommendation system216modifies one of the recommendation models using the corresponding indications of the plurality of user actions as training data in an incremental learning process. In some example embodiments, the recommendation model comprises a fixed-effects model and a random-effects model, and the incremental learning process comprises modifying the random-effects model. The modifying of the recommendation model may comprise receiving the corresponding indications of the plurality of user actions as streaming data, determining that the streaming data of the corresponding indications of the plurality of user actions satisfies a minimum threshold amount of data, and triggering the incremental learning process based on the determining that the streaming data of the corresponding indications satisfies the minimum threshold amount of data, with the corresponding indications of the plurality of user actions being used as a batch of training data in the incremental learning process. In some example embodiments, the incremental learning algorithm inFIG.9, which was previously discussed above, is used to modify the recommendation model. However, other incremental learning algorithms may be used to modify the recommendation model.

It is contemplated that any of the other features described within the present disclosure can be incorporated into the method1000.

The example computer system1100includes a processor1102(e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory1104and a static memory1106, which communicate with each other via a bus1108. The computer system1100may further include a graphics display unit1110(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system1100also includes an alphanumeric input device1112(e.g., a keyboard or a touch-sensitive display screen), a user interface (UI) navigation device1114(e.g., a mouse), a storage unit1116, a signal generation device1118(e.g., a speaker) and a network interface device1120.

The storage unit1116includes a machine-readable medium1122on which is stored one or more sets of instructions and data structures (e.g., software)1124embodying or utilized by any one or more of the methodologies or functions described herein. The instructions1124may also reside, completely or at least partially, within the main memory1104and/or within the processor1102during execution thereof by the computer system1100, the main memory1104and the processor1102also constituting machine-readable media.

Although an embodiment has been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the present disclosure. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof, show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments illustrated are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. This Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.