Rule evaluation based on precomputed results

Techniques are described for employing precomputed results of applying rules to content items, the rules applicable to determine whether content items may be electronically published. On receiving a request for a content item, rules applicable to the content item may be identified. A datastore of precomputed results of rule application may be accessed to determine whether the datastore includes a result of applying a current or previous version of each rule. If the datastore includes a current result, the current result may be employed in determine whether the content item may be presented. If the datastore includes a previous result, the previous result may be so employed. If the datastore includes a previous result or no appropriate result, a job may be queued to calculate the current result asynchronously relative to the request for the content item.

BACKGROUND

An organization engaged in software development, electronic commerce, or other types of services may generate and consume a large quantity of data regarding operations of the organization. Such data may be analyzed through the execution of any number of jobs, and the results of such jobs may be provided to data consumers who are internal to or external to the organization. Because the results of the executed jobs may be employed in subsequent operations related to marketing, inventory management, system design, personnel, and so forth, delays or failures in job execution may delay or otherwise adversely affect such subsequent operations.

Certain implementations and embodiments will now be described more fully below with reference to the accompanying figures, in which various aspects are shown. However, various aspects may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. Like numbers refer to like elements throughout.

DETAILED DESCRIPTION

This disclosure describes implementations of systems, devices, methods, and computer-readable media for employing stored, precomputed results of rule application in scenarios where rules are to be applied to determine whether content items are to be presented, or in other scenarios. In some implementations, a request may be received for a content item to be presented through an application executing on a user device, such as through a web browser. In some cases, a content server device or other computing device receiving the request may determine that a rule is to be applied to the requested content item. The rule may implement a policy or business decision as to which content items are appropriate for presentation. For example, a rule may apply a policy that a content item is not to be presented, through a web site or otherwise, if the content item includes one or more of: potentially offensive content; profane, obscene, or prurient content; sensitive or confidential personal information regarding one or more users; content that is subject to copyright, trademark, or service mark protections; and so forth. The application of a rule may include any number of operations to evaluate or otherwise analyze any number of aspects or features of a content item.

In some cases, the application of a rule may be computationally expensive in that it consumes a substantial amount of processing, storage, or networking resources on one or more computing devices. For example, a rule to identify potentially offensive, obscene, prurient, copyrighted, or confidential text information in a content item may include analyzing a content item using one or more computationally expensive text analysis operations. Such operations may include, but are not limited to, regular expression searches, natural language analyses, machine learning based operations, comparison of text elements to one or more dictionaries of disallowed terms or phrases, and so forth. As another example, a rule to identify potentially offensive, obscene, prurient, copyrighted, or confidential images in a content item such as still images or frames of a video may employ one or more computationally expensive operations including, but not limited to, pattern recognition algorithms, comparison of image elements to collections of other images, image processing operations for color modification, boundary recognition, compression, decompression, watermark detection, format conversion, resolution modification, and so forth. Moreover, in some cases the application of a rule may be computationally expensive because applying the rule includes performing operation(s) against a large number of records in a dataset.

In some cases, rule(s) may be applied substantially in real time following the receipt of a request for a content item. The application of a rule in real time, or substantially in real time, may include performing one or more operations to apply the rule within a predetermined time period after receiving the request for the content item. In some cases, the application of a rule in real time, or substantially in real time, may include launching the operation(s) to apply the rule synchronously with the request for the content item. A synchronous application of a rule may include applying the rule within a same execution path as the process that received and analyzed the request for the content item. For example, an executing process may receive and analyze the request for the content item, and the same process may itself perform the operation(s) to apply to the rule to the content item. Alternatively, the receiving process may launch, spawn or otherwise execute one or more other processes, sub-processes, or threads to begin performing the operation(s).

The rule(s) may be defined within a business or other organization that provides a web site or that otherwise delivers information over a network. In some cases, the application of a rule synchronously with, or substantially in real time following, the receipt of a request for a content item may be based on a decision that the rule is to be applied within a predetermined (e.g., short) period of time following the request. For example, an organization may host content that is posted to a web site by customers or other users, and the organization may seek to quickly identify posted content that is potentially fraudulent, potentially offensive to users, or that otherwise is in violation of a content publication policy of the organization. As another example, a business may host web sites of online merchants and may seek to quickly apply rules in time-sensitive situations, such as when merchants go on vacation, when merchants provide limited time offers of products or services, when tax rates change, when there is a holiday in a particular region, and so forth. In cases where it is computationally expensive to apply a rule to a content item, re-computing the result of rule substantially in real time or synchronously with respect to a request for the content item may delay those operations that consume or otherwise depend on the result. For example, performing a computationally expensive set of operations to apply a rule to a requested content item substantially in real time or synchronously with the request may delay the presentation of the content item, leading to a reduction in sales transaction, a degradation of user experience through a web site, or other negative effects.

Accordingly, implementations employ stored, precomputed results of rules to expedite the substantially real time or synchronous application of rules to content items. In some implementations, data storage may store precomputed results of applying one or more versions of one or more rules to content items. On receiving a request for a content item to be presented on a user device, implementations may determine whether the data storage stores a result of applying a current version of the rule to the content item. If so, that result may be retrieved and employed (e.g., substantially in real time) to determine whether the content item is to be delivered for presentation. If the data storage stores a result of applying a previous version of the rule to the content item, the stored result may be retrieved and employed (e.g., substantially in real time) to determine whether the content item is to be delivered for presentation. A job may then be added to a queue to execute asynchronously (e.g., at a later time, not in real time with the request) to compute a current result of applying the current version of the rule to the content item and to store the current result in the data storage. If the data storage does not store a result of applying any version of the rule to the content item, a default result may be employed and the rule application job may be added to the queue. Accordingly, implementations provide a technique for providing the result of applying the rule substantially in real time or synchronously with the request for a content item, while mitigating the performance impact of re-computing the result in real time.

FIG. 1depicts an environment100including one or more user devices102. The user device(s)102may comprise any type of computing device, including but not limited to a personal computer, a network computer, a cloud computing or distributed computing device, a smartphone, a tablet computer, a wearable computer, an implanted computer, a mobile gaming device, an electronic book reader, an automotive computer, a desktop computer, a laptop computer, a thin client, a terminal, a game console, a smart appliance, a home entertainment device, and so forth. Although examples herein may describe the user device(s)102as physically separate devices, implementations are not so limited. In some cases, the user device(s)102may include one or more of a virtual computing environment, a hypervisor, a device emulation, or a virtual machine executing on one or more physical computing devices.

A user device102may be employed by a user to generate and send a content item request104. The content item request104may include a content item identifier (ID)106that identifies a content item108to be presented on the user device102. In some cases, the content item request104may be generated and sent through a web browser executing on the user device102. In such cases, the content item request104may be specified according to a protocol such as a version of Hypertext Transport Protocol (HTTP), HTTP Secure (HTTPS), File Transfer Protocol (FTP), and so forth. The content item ID106may be specified as a Uniform Resource Identifier (URI), such as a Uniform Resource Locator (URL) or Uniform Resource Name (URN) that provides a network location of a page that includes the requested content item108. Alternatively, the content item request104may be generated and sent from a native application executing on the user device102, to request that a content item108be presented through a user interface of the native application. In such cases, the native application may include any type of application that executes, at least in part, outside of a web browser on the user device102.

Implementations support any type or format of content within the content item108, including but not limited to one or more of the following:

Text content, including any amount of alphanumeric or symbolic character text described using any character set or code page, such as any version or subset of the Unicode character standard. Text components may also include metadata described using a markup language, such as any version of Hypertext Markup Language (HTML), Dynamic HTML (DHTML), Extensible HTML (XHTML), Extensible Markup Language (XML), and so forth;

Script files or program modules that provide dynamic content using any scripting or programming language, including any version of JavaScript™, VBScript™, Perl™, Common Gateway Interface (CGI), Cascading Style Sheets (CSS), other types of style sheets, and so forth;

Image files in any format, such as a version of the Joint Photographic Experts Group (JPEG) format, the Graphics Interchange Format (GIF), the Bitmap (BMP) image file format, and so forth; or

Audio, video, audio-visual, or multimedia content files in any format, such as a version of the Waveform Audio File Format (WAV), the AU file format, the Moving Picture Experts Group (MPEG) formats such as MPEG-1, MPEG-2, MP3, or MPEG-4 formats, the Audio Video Interleave (AVI) format, and so forth.

In some cases, the content item request104may include multiple content item IDs106, requesting the presentation of multiple content items108on the user device102. Alternatively, the content item request104may not include a particular content item ID106, and may instead include other information to be used to identify one or more content items108to be presented on the user device102. For example, the content item request104may include a search query of one or more search terms, and may request the delivery of one or more content items108that correspond to the search query. Moreover, although examples herein may describe the content item108as web content to be presented within a web browser, implementations are not so limited. Implementations also support content items108that are presentable through other applications executing on the user device102. For example, implementations support the delivery of content items108that are presentable through a native application executing on the user device102, where the native application may be written and compiled to execute on the particular processor hardware and operating system supported by the user device102.

Implementations support any type of content in the content item(s)108that are presentable on the user device102. In some cases, a content item108may include text, image, or other data describing a product or service that is made available to users of the user device(s)102through a web site or an application executing on the user device(s)102. In some cases, the content item(s)108may include public postings to a social network web site or application, product review web site or application, public message board, weblog, and so forth. In some cases, the content item(s)108may include the results of searching the web, or searching some other information repository, based on a search query of one or more search terms. Implementations also support other types of content in the content item(s)108.

The content item request(s)104may be communicated over one or more networks to one or more content server device(s)110. The content server device(s)110may comprise any type of computing device, including but not limited to the types of computing devices described with reference to the user device(s)102. In some cases, two or more of the content server devices110may comprise a cluster, cloud, farm, or other grouping of multiple devices that coordinate operations to provide load balancing, failover support, parallel processing capabilities, shared storage resources, or other aspects. Although examples herein may describe the content server device(s)110as physically separate devices, implementations are not so limited. In some cases, the content server device(s)110may include one or more of a virtual computing environment, a hypervisor, a device emulation, or a virtual machine executing on one or more physical computing devices. The content server device(s)110are described further with reference toFIG. 4.

The content server device(s)110may execute a content service module112. The content service module112may include any number of web servers, application servers, or other software modules that are configured to serve content items108. In some cases, the content server device(s)110may cache one or more recently requested content items108in local storage on the content server device(s)110, or in storage that is accessible to the content server device(s)110. In such cases, if the requested content item(s)108are available in the cache, the content service module112may retrieve and serve the requested content item(s)108from the cache.

The content service module112may communicate the content item ID(s)106over one or more networks to one or more rule application devices114. The rule application device(s)114may comprise any type of computing device, including but not limited to the types of computing devices described with reference to the user device(s)102. In some cases, two or more of the rule application devices114may comprise a cluster, cloud, farm, or other grouping of multiple devices that coordinate operations to provide load balancing, failover support, parallel processing capabilities, shared storage resources, or other aspects. Although examples herein may describe the rule application device(s)114as physically separate devices, implementations are not so limited. In some cases, the rule application device(s)114may include one or more of a virtual computing environment, a hypervisor, a device emulation, or a virtual machine executing on one or more physical computing devices. The rule application device(s)114are described further with reference toFIG. 5.

The content item ID(s)106may be received by a content item evaluation module116executing on the rule application device(s)114. The content item evaluation module116may analyze the received content item ID(s)106and determine whether any rule(s)118are to be applied to any of the content item(s)108identified by the content item ID(s)106. The content item evaluation module116may also access rule version data120listing a current version of each of the rule(s)118to be applied to the requested content item108. In some cases, the content item evaluation module116may determine a type of the content item108requested and identify the rule(s)118to be applied to that type of content item108. For example, a particular set of one or more rules118may be designated to be applied to a content item108if the content item108includes a description of a product offered through an online store. As another example, a particular set of one or more rules118may be designated to be applied to a content item108if the content item108is a public posting on a social network web site, or if the content item108is a search result produced by a search service. In some cases, all of the rules118may be applied to each content item108requested. In some cases, the rules118may be defined or provided by one or more individuals or organizations to implement particular policies regarding the presentation of the content item(s)108through a web site, application, or otherwise on the user device(s)102.

The content item evaluation module116may access result data122. As shown inFIG. 1, the result data122may be stored in local data storage on the rule application device(s)114. Alternatively, the result data122may be stored externally to the rule application device(s)114in data storage that is accessible to the rule application device(s)114over one or more networks. The result data122may include the stored results of one or more previous applications of the rule(s)118to content items108. The result data122is described further with reference toFIGS. 2 and 3. For each rule118to be applied to the content item108corresponding to the content item ID106, the content item evaluation module116may search the result data122to determine whether the result data122includes a precomputed (e.g., previously computed) result of applying a version of the rule118to the content item108.

In some implementations, one or more of the rule(s)118may be updated to apply a new policy governing content presentation or for other reasons. Accordingly, there may be multiple versions of a rule118, including a current (e.g., latest, most recently updated) version of the rule118and zero or more previous versions of the rule118. For each pairing of a content item108and a rule118that is applicable to the content item108, the result data122may store a current result that is the result of applying the current version of the rule118to the content item108. For each pairing of a content item108and a rule118applicable to the content item108, the result data122may also store one or more previous results, each previous result being the result of applying a previous version of the rule118to the content item108. The current version(s) of each of the rule(s)118may be described in the rule version data120, which may list for each rule118a version ID denoting the current version of the rule118.

For each of the rule(s)118to be applied to the requested content item108, the content item evaluation module116may determine whether the result data122stores a result of applying a current version of the rule118to the content item108. If the result data122includes a precomputed result of applying a current version of the rule118to the content item108, this current result may be retrieved and employed by modules executing on the content server device(s)110to determine whether the content item108is to be delivered for presentation on the user device102. In some implementations, the result data122may identify the version of the rule118used to generate each of the results in the result data122. The content item evaluation module116may compare the rule version included in the result data122to the rule version data120listing the current version of one or more rules118. If the two versions are substantially the same or otherwise correspond, the content item evaluation module116may determine that the result data122includes a current result of applying a current version of the rule118to the content item108.

In some cases, the result data122may include a previous result of applying a previous version of the rule118to the content item108. In such cases, the previous result may be retrieved from the result data122and employed by modules executing on the content server device(s)110to determine whether the content item108is to be delivered for presentation on the user device102. In cases where the rule118is substantially static, e.g., such that it changes incrementally from version to version, the previous result may provide an approximation of a current result when a current result is not present in the result data122. In cases where the result data122does not include a current result or a previous result of applying a current version or a previous version of the rule118to the content item108, a default result may be employed in determining whether to deliver the content item108. In some implementations, the default result may be a Boolean true value, e.g., indicating that the content item108is to be delivered for presentation on the user device102.

If the result data122does not store a current result of applying the rule118, such that a previous result or a default result is employed, the content item evaluation module116may add a job to a job queue124. The job queue124may include a listing of one or more jobs that are each executable to compute a current result of applying a current version of a rule118to a content item108. Adding the job to the job queue124may include queuing the job by placing it at the top of the job queue124, the bottom of the job queue124, or into any other location in the job queue124. The jobs in the job queue124may be executed by a results computation module126executing on the rule application device(s)114. The results of executing the jobs may be added to the result data122.

In some implementations, the jobs in the job queue124may be executed asynchronously with the receipt of the content item request(s)104, such that each of the jobs executes at a time that is independent of the time(s) when the content item request(s)104are received and processed. In some implementations, the timing of the execution of the jobs may depend on resource availability on the rule application device(s)114, e.g., the jobs may execute when processor usage for other processes is below a predetermined threshold usage. Alternatively, the jobs in the job queue124may be executed at a scheduled day and time. In some cases, the jobs in the job queue124may be executed at a predetermined frequency, e.g., nightly, hourly, and so forth. AlthoughFIG. 1depicts the results computation module126as executing on a same set of rule application device(s)114as the content item evaluation module116, in some implementations the results computation module126may execute on separate computing device(s) relative to the content item evaluation module116. The processing of the jobs in the job queue124is described further with reference toFIG. 9.

As shown inFIG. 1, the rule application device(s)114and the content server device(s)110may access one or more content storage system(s)128that store the content item(s)108. The content storage system(s)128may provide data storage on any number of databases, datastores, or other data storage devices. The data storage may comprise any number of tables, arrays, structured lists, trees, or other data structures. The content storage system(s)128may provide data storage in any relational or non-relational data storage format. Although the content storage system(s)128are depicted as external to the rule application device(s)114and the content server device(s)110, implementations are not so limited. In some implementations, the content storage system(s)128may be at least partly incorporated into one or both of the rule application device(s)114or the content server device(s)110.

In some implementations, the results computation module126may be configured to modify or remove a content item108in the content storage system(s)128based on the results of applying one or more of the rule(s)118to the content item108. For example, if the results computation module126computes a result of applying a rule118to a content item108, and the result indicates that the content item108is not to be presented on the user device(s)102, then the results computation module126may remove the content item108from the content storage system(s)128. Alternatively, the results computation module126may modify the stored content item108to indicate that the content item108may not be presented, e.g., by adding a flag or other data to the record for the content item108in the content storage system(s)128. In some implementations, the result data122may be accessed by a content aggregation module130executing on the content server device(s)110. The content aggregation module130may be configured to collect information from any number of source devices, processes, or databases regarding content items108that are presentable on user devices102. For example, the content aggregation module130may be configured to access the content storage system(s)128and retrieve the content item(s)108stored therein. The content aggregation module130may correlate, interpret, or transform the information to prepare pages (e.g., web pages) that include the content item(s)108and that are presentable (e.g., through a web browser) on the user device(s)102. In some cases, the content aggregation module130may transform information regarding the content item(s)108into a summarized (e.g., abstracted) format that may be more efficiently communicated to other devices compared to the raw data of the content item(s)108. In some implementations, the content aggregation module130may employ the results stored in the result data122to determine which content item(s)108may be aggregated or otherwise processed to be delivered to the user device(s)102for presentation. If the result data122includes a result that indicates a content item ID106is not to be presented, the content aggregation module130may not provide the corresponding content item108for presentation on the user device(s)102.

In some implementations, the result data122may be accessed by a search module132executing on the content server device(s)110. The search module132may be configured to generate search results in response to a search query received from a user device102, where such search results may include one or more content items108. In some implementations, the search module132may omit a content item108from a list of provided search results if the result data122includes a result indicating that the content item is not to be presented.

Based on the result data122, one or both of the content aggregation module130or the search module132may provide one or more content items108to the content service module112. The content service module112may then serve the content item(s)108to the user device(s)102in response to the content item request(s)104. The delivery of content item(s)108based on the results of applying the rule(s)118is described further with reference toFIGS. 6-8. In some implementations, the result data122may be employed to determine whether a content item108is to be provided to a user device102in response to a content item request104. Such implementations are described further with reference toFIG. 7. In some implementations, the result data122may be employed to determine, adjust, or modify the presentation of the content item108in response to the content item request104. Such implementations are described further with reference toFIG. 8.

In some implementations, a content item108may be provided in response to a content item request104, and that content item request104may trigger the analysis to determine whether the content item108complies with one or more rules118. The result data122resulting from the application of the rule(s)118may then be employed to determine whether the content item108is to be provided in response to subsequent content item requests104. In some cases, the result data122may be employed to determine whether the content item108is to be subsequently listed in a catalog of content items108(e.g., a product catalog) that is assembled by the content aggregation module130for presentation to a user. The result data122may also be employed to determine whether the content item108is to be subsequently included in search results generated by the search module132for presentation to a user. Accordingly, although the content item108may be presented in response to the first content item request104for that content item108, the results of applying the rule(s)118may be employed to determine whether the content item108is to be subsequently published, exposed, or otherwise presented to users. In some implementations, a result of applying a rule118to a content item108may persist in the result data122without expiration until an updated version of the rule118prompts a re-computation of the result for the content item108.

In some implementations, if the current result, previous result, or default result is a Boolean false value, e.g., indicating that the content item108is not to be delivered for presentation on the user device102, then the content item evaluation module116may add the content item ID106to a blacklist. The blacklist may include zero or more content item IDs106to indicate those content items108to be blocked from presentation on user devices102. Alternatively, some implementations may employ a whitelist instead of the blacklist. The whitelist may list those content items108that may be presented on user devices102, and the whitelist may indicate the blocked content items108based on their omission from the whitelist. Implementations support the use of any number or any combination of whitelists and blacklists to indicate those content items108that may or may not be presented on user devices102. In some implementations, the content item ID(s)106may persist in the blacklist without expiration until an updated version of the rule(s)118prompts a re-evaluation of the results for the content item(s)108. The blacklist(s) or whitelist(s) may be employed by one or both of the content aggregation module130or the search module132to determine whether to present the content item(s)108.

Implementations may enable an expedited response to changes in externally imposed (e.g., government) regulations or to changes in internal business policies regarding the presentation of content items108. In response to such changes, one or more of the rules118may be updated to prevent, allow, or modify the presentation of the content item(s)108in view of the changed regulations or polices. As the content item(s)108are requested, the updated versions of the rule(s)118may be applied to update the result data122. The result data122may then be employed to determine which content item(s)108may be presented or to determine a manner of presenting the content item(s)108.

In some implementations, the receipt of a content item request104for a content item108may trigger the accessing of the result data122to determine the presentation of the content item108. Accessing the result data122and determining the presentation of the content item108may be performed substantially synchronously with the content item request104. In cases where the result data122does not already include the precomputed result(s) of applying current version(s) of the rule(s)118to the requested content item108, one or more jobs may be added to the job queue124to compute the result(s) asynchronously relative to the content item request104. Accordingly, implementations support the computation of result(s) for those content item(s)108that are requested for presentation.

Implementations also support the application of the rule(s)118to compute result(s) based on other types of events. In some implementations, the application of one or more rules118to a content item108may be substantially synchronous with receiving the content item108to be stored in the content storage system(s)128. In some cases, the content item(s)108may be received from end users, merchants whose web sites are served through the content server device(s)110, or other third party content providers. In such cases, implementations enable the rule(s)118to be applied to the content item108when the content item108is ingested into the system, e.g., received to be stored in the content storage system(s)128. If the result(s) of applying the rule(s)118indicate that the content item108is not to be presented on the user device(s)102, the content item108may not be stored in the content storage system(s)128. Alternatively, if the result(s) indicate that the content item108is not to be presented, the content item108may be stored but with a flag or other information indicating that it is not to be presented to users. In some cases, the rule(s)118applied during ingestion of the content item(s)108may be different than the rule(s)118applied in the jobs that are added to the job queue124based on the content item request(s)104.

In some implementations, the rule(s)118may be applied to one or more of the content items108stored in the content storage system(s)128independently of the content item request(s)104for the content item(s)108. For example, a process may execute one or more times as a scheduled process, or in response to an operator request, to apply the rule(s)118to stored content item(s)108. The result data122generated based on the application of the rule(s)118may then be subsequently used to determine whether to present the content item(s)108in response to the content item request(s)104. Alternatively, the generated result data122may be employed to determine which of the content item(s)108, if any, are to be removed from the content storage system(s)128or flagged for non-presentation. In some implementations, the rule(s)118applied during such a process (e.g., a static or background process) may be different than the rule(s)118applied (e.g., dynamically) in the jobs that are added to the job queue124based on the content item request(s)104. Moreover, in some implementations the rule(s)118applied to the content item(s)108may differ based on characteristics of the content item request104. For example, the applied rule(s)118may be based at least partly on one or more of the following: a geographic origin of the content item request104that prompted the application of the rule(s)118; a type of the user device102that sent the content item request104(e.g., whether the user device102is a mobile device or otherwise); information regarding the user that generated the content item request104(e.g., age of the user, the user's language preference, and so forth); or other characteristics of the content item request104.

Although some examples herein describe implementations in which the precomputed results of applying the rule(s)118may be employed to determine whether one or more particular content items108are to be delivered for presentation on user device(s)102, implementations are not so limited. In some implementations, one or more rules118may be applied asynchronously to other types of data item(s) to generate stored result data122. A stored result may then be accessed and provided in response to a request for the result of applying a rule118to an input data item, instead of computing the result in real time (e.g., dynamically) in response to the request. In this way, implementations enable the result(s) to be provided more efficiently and may thus mitigate the delay that may be caused by re-computing a result substantially in real time. Such implementations are described further with reference toFIG. 10.

In some cases, it may be desirable to expedite the evaluation of a function y=f(x) based on an input data item x, and the computational cost of evaluating f(x) may be sufficiently high such that it may be disadvantageous to compute f(x) substantially in real time in response to a request for y given a large dataset to be accessed to evaluate y. Moreover, in some cases the definition of f(x) may be slowly changing from version to version, and there may exist a default result for f(x) (e.g., a Boolean true result) that is usable, in many cases, by the process requesting y. In such cases, the implementations described herein may enable a usable result y to be provided substantially in real time while the potentially more accurate result y is computed in the background or otherwise asynchronously with respect to the request for y.

The computation of the results of evaluating the rule(s)118with respect to content item(s)108may be triggered in response to the content item request(s)104for the particularly requested content item(s)108, and the results may be stored for those particularly requested content item(s)108. Accordingly, implementations may be described as providing a lazy evaluation and storage of those results that are requested. The combination of such a lazy evaluation technique and the lookup of the stored results may enable implementations to provide results efficiently in response to requests, and the results may be sufficiently accurate in cases where the rule(s)118change incrementally (e.g., slowly) from version to version.

The various devices of the environment100may communicate with one another using one or more networks. Such networks may include public networks such as the Internet, private networks such as an institutional or personal intranet, or some combination of private and public networks. The networks may include any type of wired or wireless network, including but not limited to local area networks (LANs), wide area networks (WANs), wireless WANs (WWANs), wireless LANs (WLANs), mobile communications networks (e.g. 3G, 4G, etc.), and so forth. In some implementations, communications between the various devices in the environment100may be encrypted or otherwise secured. For example, such communications may employ one or more public or private cryptographic keys, ciphers, digital certificates, or other credentials supported by a security protocol such as any version of the Secure Sockets Layer (SSL) or the Transport Layer Security (TLS) protocol.

FIG. 2depicts a schematic200illustrating an example format for the result data122, including the result(s) of previous applications of the rule(s)118to the content item(s)108or to other types of input data items. As shown inFIG. 2, the result data122may include any number of records (e.g., rows). Each record may include the content item ID106identifying the content item108to which a rule118was applied. Each record may also include a rule version ID202describing a version of the rule118applied to the content item108. The rule version ID202may be numeric (e.g., version 1, version 2.1, etc.). The rule version ID202may also include a timestamp, such as a date and time when the applied version of the rule118was created or otherwise became available. Each record of the result data122may include the result204of applying the version of the rule118specified by the rule version ID202to the content item108identified by the content item ID106. In some implementations, each record of the result data122may also include a rule description206that describes the rule118applied. In some implementations, the rule118may be a machine readable portion of script or code that is executable to apply the rule118to a content item108. In some cases, the result data122may include a rule ID that may be employed as a key to retrieve the rule118from a separate table that stores the rule(s)118.

FIG. 3depicts a schematic300illustrating an example of the result data122including the result(s)204of previous applications of the rule(s)118. As shown in the example ofFIG. 3, the result data122may include any number of records, rows, or data elements that each correspond to a result204of applying a rule118to a content item108, or to some other type of input data item. Although the example ofFIG. 3depicts the result data122described using a markup language such as a version of the Extensible Markup Language (XML), implementations are not so limited. Implementations support the use of any format to describe the result data122, including formats that employ or do not employ metadata tags to identify particular data elements. In the example ofFIG. 3, each record of result data122is described in a tag, and the content item ID106, the rule118, the rule version ID202, the result204, and the rule description206are each included as attributes in the tag. For example, as shown inFIG. 3, each of these attributes may be respectively indicated by tags “content_item_ID”, “rule_ID”, rule_version”, “rule_description”, and “result”. Alternatively, one or more of the attributes may be included in the result data122as a separate tag between open and close tags such as an <result_data> tag and a </result_data> tag.

AlthoughFIG. 3depicts particular example formats and data for the content item ID106, rule118, rule version ID202, result204, and rule description206, implementations are not limited to these examples. Moreover, implementations are not limited by the example order and format of the result data122shown inFIG. 3. Further, although examples herein describe the results204as Boolean true or false values, indicating that a content item108may be presented or may not be presented, implementations also support the use of other data types for the results204. For example, implementations may support the employment of rules118that are configured to classify one or more content items108into a category, class, sub-class, product type, brand, seller, price, or other characteristic. In such cases, the result(s)204may be numeric data, text strings, character data, and so forth.

AlthoughFIGS. 2 and 3depict examples in which a particular set of data attributes are included in a record corresponding to a result204, implementations are not limited to these examples. In some implementations, the rule description206may be stored in a separate table from the result data122. In such cases, a rule ID may be employed as a secondary key to access the separate table and to retrieve the rule description206for a particular rule118.

FIG. 4depicts a block diagram400of an example of the content server device(s)110. As shown in the block diagram400, the content server device(s)110may include one or more processors402(e.g., hardware-based processor(s)) configured to execute one or more stored instructions. The processor(s)402may comprise one or more cores.

The content server device(s)110may include one or more input/output (I/O) devices404. The I/O device(s)404may include input devices such as a keyboard, a mouse, a pen, a game controller, a touch input device, an audio input device (e.g., a microphone), a gestural input device, a haptic input device, an image or video capture device (e.g., a camera), or other devices. In some cases, the I/O device(s)404may also include output devices such as a display, an audio output device (e.g., a speaker), a printer, a haptic output device, and so forth. The I/O device(s)404may be physically incorporated with the content server device(s)110, or may be externally placed.

The content server device(s)110may include one or more I/O interfaces406to enable components or modules of the content server device(s)110to control, interface with, or otherwise communicate with the I/O device(s)404. The I/O interface(s)406may enable information to be transferred in or out of the content server device(s)110, or between components of the content server device(s)110, through serial communication, parallel communication, or other types of communication. For example, the I/O interface(s)406may comply with a version of the RS-232 standard for serial ports, or with a version of the Institute of Electrical and Electronics Engineers (IEEE) 1284 standard for parallel ports. As another example, the I/O interface(s)406may be configured to provide a connection over Universal Serial Bus (USB) or Ethernet. In some cases, the I/O interface(s)406may be configured to provide a serial connection that is compliant with a version of the IEEE 1394 standard. The content server device(s)110may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components of the content server device(s)110.

The content server device(s)110may include one or more network interfaces408that enable communications between the content server device(s)110and other network accessible computing devices, such as the user device(s)102or the rule application device(s)114. The network interface(s)408may include one or more network interface controllers (NICs) or other types of transceiver devices configured to send and receive communications over a network.

The content server device(s)110may include one or more memories, described herein as memory410. The memory410comprises one or more computer-readable storage media (CRSM). The CRSM may include one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memory410provides storage of computer-readable instructions that may describe data structures, program modules, processes, applications, or other data for the operation of the content server device(s)110. In some implementations, the memory410may provide storage of computer-readable instructions or other information in a non-transitory format.

The memory410may include an operating system (OS) module412. The OS module412may be configured to manage hardware resources such as the I/O device(s)404, the I/O interface(s)406, and the network interface(s)408, and to provide various services to applications, processes, or modules executing on the processor(s)402. The OS module412may include one or more of the following: any version of the Linux™ operating system; any version of iOS from Apple™ Corp. of Cupertino, Calif., USA; any version of Windows™ or Windows Mobile™ from Microsoft™ Corp. of Redmond, Wash., USA; any version of Android™ from Google™ Corp. of Mountain View, Calif., USA and its derivatives from various sources; any version of Palm OS™ from Palm Computing™, Inc. of Sunnyvale, Calif., USA and its derivatives from various sources; any version of BlackBerry OS™ from Research In Motion™ Ltd. of Waterloo, Ontario, Canada; any version of VxWorks™ from Wind River Systems™ of Alameda, Calif., USA; or other operating systems.

The memory410may include one or more of the modules described above as executing on the content server device(s)110, such as one or more of the content service module112, the content aggregation module130, or the search module132. The memory410may also include one or more other modules414, such as a user authentication module or an access control module to secure access to the content server device(s)110, and so forth.

The memory410may include data storage416to store data for operations of the content server device(s)110. The data storage416may comprise a database, array, structured list, tree, or other data structure, and may be a relational or a non-relational datastore. The data storage416may store data such as that described above, including one or more of the content item request(s)104, the content item ID(s)106, or the content item(s)108. The data storage416may also store other data418, such as user authentication information or access control data. In some implementations, at least a portion of the information stored in the data storage416may be stored externally to the content server device(s)110, on other devices that may communicate with the content server device(s)110via the I/O interface(s)406or via the network interface(s)408.

FIG. 5depicts a block diagram500of an example of the rule application device(s)114. As shown in the block diagram500, the rule application device(s)114may include one or more processors502(e.g., hardware-based processor(s)) configured to execute one or more stored instructions. The processor(s)502may comprise one or more cores. The rule application device(s)114may include one or more I/O devices504, one or more I/O interfaces506, and one or more network interfaces508as described above respectively with reference to the I/O device(s)404, the I/O interface(s)406, and the network interface(s)408.

The rule application device(s)114may include one or more memories, described herein as memory510. The memory510comprises one or more CRSM, as described above with reference to the memory410. The memory510may include an OS module512that is configured to manage hardware resources such as the I/O device(s)504, the I/O interface(s)506, and the network interface(s)508, and to provide various services to applications, processes, or modules executing on the processor(s)502. The OS module512may include one or more of the operating systems described above with reference to the OS module412. The memory510may include one or more of the modules described above as executing on the rule application device(s)114, such as the content item evaluation module116and the results computation module126. The memory510may also include one or more other modules514, such as a user authentication module or an access control module to secure access to the rule application device(s)114, and so forth.

The memory510may include data storage516, which may store data for operations of the rule application device(s)114. The data storage516may comprise a database, array, structured list, tree, or other data structure, and may be a relational or a non-relational datastore. The data storage516may include data that is in active memory on the rule application device(s)114, or data that is written to a hard drive, disk, or other non-volatile storage on the rule application device(s)114. The data storage516may store data such as that described above, including one or more of the rule(s)118, the result data122, or the job queue124. In implementations that employ blacklist(s) or whitelist(s) to determine whether to present content item(s)108, the data storage516may also store such blacklist(s) or whitelist(s).

In some implementations, the data storage516may also store the rule version data120. The rule version data120may list the current rule version ID202for one or more rules118. The rule version data120may be employed by the content item evaluation module116to determine whether the result data122includes a current result of applying a current version of a rule118to a content item108, a previous result of applying a previous version of a rule118to a content item108, or no result of applying any version of a rule118. The data storage516may also store other data518, such as user authentication information or access control data. In some implementations, at least a portion of the information stored in the data storage516may be stored externally to the rule application device(s)114, on other devices that may communicate with the rule application device(s)114via the I/O interface(s)506or via the network interface(s)508.

FIG. 6depicts a flow diagram600of a process for retrieving a stored result204in response to the content item request104for the content item108, if an appropriate stored result204is available in the result data122, and in some cases queuing a job to compute a current result if the current result is not stored. One or more operations of the process may be performed by the content service module112, the content aggregation module130, the search module132, other modules executing on the content server device(s)110, the content item evaluation module116, the results computation module126, other modules executing on the rule application device(s)114, or modules executing on other devices.

At602, a content item request104is received or an indication of a content item request104is received. The content item request104may include the content item ID106, and may request that the content item108corresponding to the content item ID106be delivered for presentation on the user device102that sent the content item request104.

At604, one or more rules118may be identified that are applicable to determine whether the requested content item108is to be delivered for presentation on the user device102, and a current version of each of the rule(s)118may be determined based on the rule version data120. In some implementations, the identification of the rule(s)118and the determination of the current version(s) may be performed synchronously with the content item request104.

At606, the result data122may be accessed from the data storage516. As described above, the result data122may store precomputed (e.g., previously computed) results204of applying the rule(s)118to content items108and each of the results204may indicate whether a corresponding content item108may be presented on the user device(s)102. In some implementations, the access to the result data122may be performed synchronously with receiving the content item request104. Moreover, in some cases one or more of the subsequent operations608,610,612,614,616, or618may also be performed synchronously relative to receiving the content item request104.

At608, a determination is made whether the result data122stores a current result of applying a current version of the rule(s)118to the requested content item108. If so, the process may proceed to610and employ the current result to determine whether to present the content item108. After610, the process may proceed as described with reference toFIG. 7 or 8below. If the result data122does not store a current result, the process may proceed to612.

At612, a determination is made whether the result data122stores a previous result of applying a previous version of the rule(s)118to the requested content item108. If so, the process may proceed to614and employ the previous result to determine whether to present the content item108. After614, the process may proceed to618. If the result data122does not store a previous result, the process may proceed to616. As described above, in some implementations the determination of whether the result data122includes a current result or a previous result may be made by comparing the rule version ID202listed in the rule version data120(e.g., the current version of the rule118) to the rule version ID202listed in the portion of the result data122corresponding to each of the rule(s)118applied to the requested content item108.

At616, if the result data122does not include a current result or a previous result of applying the rule(s)118to the content item108, a default result may be employed to determine whether to present the content item108. In some cases, the default result may be a Boolean true value, indicating that the content item108may be delivered for presentation on the user device102.

At618, one or more jobs may be added to the job queue124. Each job may be executable to compute the current result of applying the current version of one of the rule(s)118to the content item108. As described further with reference toFIG. 9, the jobs in the job queue124may be executed asynchronously with the content item request104, at a time that is independent of the time when the content item request104is received. The process may then proceed as described with reference to eitherFIG. 7orFIG. 8.

FIG. 7depicts a flow diagram700of a process for employing the retrieved result204to determine whether to present the requested content item108in response to a received content item request104. One or more operations of the process may be performed by the content service module112, the content aggregation module130, the search module132, other modules executing on the content server device(s)110, the content item evaluation module116, the results computation module126, other modules executing on the rule application device(s)114, or modules executing on other devices.

At702, a determination is made whether the current result retrieved at610, the previous result retrieved at614, or the default result determined at616indicates that the content item108is to be delivered for presentation on the user device102that sent the content item request104. If so, at704the content item108may be provided to the user device102in response to the content item request104. If not, at706the content item108may not be provided in response to the content item request104. In implementations that employ a blacklist as described above, at706a content item108that was previously added to the blacklist may be removed from the blacklist if the current result, previous result, or default result indicates that the content item108may be presented. In implementations that employ a whitelist as described above, at706a content item108that was previously omitted from the whitelist may be added to the whitelist if the current result, previous result, or default result indicates that the content item108may be presented.

At708, in some implementations one or more additional processes or workflows may be launched based on the non-presentation of the content item108. For example, a process may be started to remove the content item108from any caches or other locations in memory. In some cases at708, a process may be started to examine clickstream data to generate a report that includes metrics or other information regarding the content item request104for the content item108, such as the number, frequency, or origin of requests received for the content item108.

FIG. 7depicts implementations in which the stored results204may be employed to determine whether to provide the content item108in response to a content item request104for the content item108. In some implementations, the content item108may be provided for presentation in response to the initial content item request104for the content item108, and subsequent presentations of the content item108may be allowed or disallowed based on the result(s)204of applying rule(s)118to the content item108through asynchronously executing (e.g., background or batch) jobs to compute the result(s)204.

FIG. 8depicts a flow diagram800of a process for employing the retrieved result204to determine a manner for presenting the requested content item108. One or more operations of the process may be performed by the content service module112, the content aggregation module130, the search module132, other modules executing on the content server device(s)110, the content item evaluation module116, the results computation module126, other modules executing on the rule application device(s)114, or modules executing on other devices.

At802, a determination is made whether the current result retrieved at610, the previous result retrieved at614, or the default result determined at616indicates that the presentation of the content item108is to be altered.

If it is determined at802that the presentation of the content item108is not to be altered, the process may process may proceed to804. At804, the content item ID106corresponding to the content item108may be provided for presentation on the user device(s)102in a manner that is unaltered from a default presentation for the content item108. For example, the default presentation of the content item108may include no additional information presented with the content item108, no alteration of the content of the content item108, or no omission of the content of the content item108.

If it is determined at802that the presentation of the content item108is to be altered, the process may proceed to806. At806, the content item108may be provided for presentation on the user device(s)102in a manner that is altered from its default presentation state. For example, in cases where the applied rule(s)118determine that the content item108includes prurient, obscene, or potentially offensive content, the presentation of the content item108may be altered to include descriptive (e.g., warning) text, added color elements (e.g., a red border), or other additional information. Alternatively, the presentation of the content item108may be modified to alter or omit the prurient, obscene, or potentially offensive content. As another example, in cases where the applied rule(s)118determine that the content item108includes copyrighted or trademarked content, the presentation of the content item108may be altered to remove such content or to incorporate the appropriate copyright or trademark designations.

At808, in some implementations one or more additional processes or workflows may be launched based on the altered presentation of the content item108. For example, a process may be started to remove the content item108from any caches or other locations in memory. In some cases at808, a process may be started to examine clickstream data to generate a report that includes metrics or other information regarding the content item request104for the content item108, such as the number, frequency, or origin of requests received for the content item108.

FIG. 9depicts a flow diagram900of a process for managing the job queue124. One or more operations of the process may be performed by the content service module112, the content aggregation module130, the search module132, other modules executing on the content server device(s)110, the content item evaluation module116, the results computation module126, other modules executing on the rule application device(s)114, or modules executing on other devices.

At902, the job queue124is accessed. As described above, the job queue124may include one or more jobs that are each executable to compute a result204of applying a current version of a rule118to a content item108. The job queue may be configured such that the jobs may be executed asynchronously with respect to the receipt of one or more content item requests104.

At904, in some implementations one or more duplicate jobs may be removed from the job queue124, to prevent the execution of duplicate and redundant jobs to calculate results204.

At906, a job is selected from the job queue124, the job describing a rule118to be applied to a content item108. Implementations support the use of any algorithm to select a job to be executed from the job queue124. In some cases, the job may be selected based on the order in which the jobs were queued (e.g., added to the job queue124). For example, the jobs may be executed in a first in, first out order in which earlier queued jobs are executed earlier than other jobs. As another example, the jobs may be executed in a last in, first out order in which the later queued jobs are executed earlier than other jobs. In some cases, the job may be selected based on the frequency or number of requests for the job that were included in the job queue124prior to the de-duplication operation at904. For example, a job that has a larger number of instances in the job queue124may be executed with priority relative to other jobs that have a smaller number of instances in the job queue124. In some cases, a more recently submitted job may be selected first for execution, to reduce a number of instances in which a potentially inappropriate content item108may be presented to users.

At908, the selected job may be executed to compute a current result of applying a current version of a rule118to the content item108specified in the selected job. At910, the computed result204may be stored in the result data122in the data storage516. After it is executed, the job may be removed from the job queue124.

At912, in some implementations the content item108may be removed from the content storage system(s)128to prevent the content item108from being presented in response to subsequent requests for the content item108. Alternatively, the record for the content item108in the content storage system(s)128may be flagged or otherwise modified to indicate that the content item108is not to be presented.

At914, a determination is made whether there are additional job(s) to be executed in the job queue124. If so, the process may return to906and select another job to be executed. If not, the process may continue to916and continue monitoring for additional jobs added to the job queue124.

Implementations also support the application of the rules118asynchronously to data item(s) of a type other than the content item(s)108described above to generate the stored result data122.FIG. 10depicts a flow diagram1000of a process for employing a stored result204in response to a request for a result of applying a rule118to a data item, if an appropriate stored result204is available, and in some cases queuing a job to compute a current result if the current result of applying a current version of the rule118is not otherwise available. One or more operations of the process may be performed by the content service module112, the content aggregation module130, the search module132, other modules executing on the content server device(s)110, the content item evaluation module116, the results computation module126, other modules executing on the rule application device(s)114, or modules executing on other devices.

At1002, a request is received, or an indication of a request is received, the request being a request for a result204of applying a rule118to an input data item, e.g., a request for the result y that is a result of calculating y=f(x) for a rule f based on input data item x. Such a request may be received from a process running on any computing device.

At1004, a rule118may be identified that is applicable to generate the result when applied to the input data item. In some implementations, the identification of the rule118may be performed substantially in real time with the receipt of the request for the result204.

At1006, the result data122may be accessed from the data storage516. The result data122may store precomputed (e.g., previously computed) results204of applying rules118to data items. In some implementations, the access to the result data122may be performed substantially in real time relative to receiving the request for the result204. Moreover, in some cases one or more of the subsequent operations1008,1010,1012,1014,1016, or1018may also be performed substantially in real time relative to receiving the request.

At1008, a determination is made whether the result data122stores a current result of applying a current version of the rule118to the input data item. If so, the process may proceed to1010. At1010, the current result may be retrieved from the result data122and provided in response to the request. If the result data122does not store a current result, the process may proceed to1012.

At1012, a determination is made whether the result data122stores a previous result of applying a previous version of the rule118to the input data item. If so, the process may proceed to1014. At1014, the previous result may be retrieved from the result data122and provided in response to the request. After1014, the process may proceed to1018. If the result data122does not store a previous result, the process may proceed to1016. As described above, in some implementations the determination of whether the result data122includes a current result or a previous result may be made by comparing the rule version ID202listed in the rule version data120(e.g., the current version of the rule118) to the rule version ID202listed in the portion of the result data122corresponding to the rule118to be applied and the data item.

At1016, a default result may be determined if the result data122does not include a current result or a previous result of applying the rule118to the input data item.

At1018, a job may be added to the job queue124. The queued job may be executable to compute the current result of applying the current version of the rule118to the data item. The job queue124may be managed as described with reference toFIG. 9.

Those having ordinary skill in the art will readily recognize that certain steps or operations illustrated in the figures above may be eliminated, combined, or performed in an alternate order. Any steps or operations may be performed serially or in parallel. Moreover, the methods described above may be implemented as one or more software programs for a computer system and may be encoded in a computer-readable storage medium as instructions executable on one or more processors.

Embodiments may be provided as a computer program product including one or more non-transitory computer-readable storage media having stored thereon instructions (in compressed or uncompressed form) that may be used to program a computer (or other electronic device) to perform processes or methods described herein. The computer-readable storage media may include one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, and so forth. For example, the computer-readable storage media may include, but are not limited to, hard drives, floppy diskettes, optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), flash memory, magnetic or optical cards, solid-state memory devices, or other types of physical media suitable for storing electronic instructions. Further, embodiments may also be provided as a computer program product including a transitory machine-readable signal (in compressed or uncompressed form). Examples of machine-readable signals, whether modulated using a carrier or unmodulated, include but are not limited to signals that a computer system or machine hosting or running a computer program may be configured to access, including signals transferred by one or more networks. For example, a transitory machine-readable signal may comprise transmission of software by the Internet.

Additionally, those having ordinary skill in the art readily recognize that the techniques described above can be utilized in a variety of devices, environments, and situations. Although the present disclosure is written with respect to specific embodiments and implementations, various changes and modifications may be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications that fall within the scope of the appended claims.