Automated decision platform

A computer system receives a set of user-defined rules are that useable by a computer service to automate a decision flow. The computer system generates a graph model from the user-defined rules. From the graph model, the computer system determines an input dependency model that is indicative of a set of inputs referred to in the graph model. The input dependency model is useable by an orchestrator to coordinate accesses to the one or more data stores in which the set of inputs is stored. The computer system receives the set of inputs and determines one or more automated decisions by applying the set of inputs to the graph model.

RELATED APPLICATIONS

The present application claims priority to PCT Appl. No. PCT/CN2020/089187, filed May 8, 2020, which is incorporated by reference herein in its entirety.

BACKGROUND

Technical Field

This disclosure relates generally to automated decision flows based on user-defined rules, and more particularly to optimizing execution of automated rules according to various embodiments.

Description of the Related Art

Computer-implemented automated decision processes are used to automate decisions according to rules defined by users. The decisions made by the computer system can subsequently be evaluated (e.g., for accuracy, for reliability, etc.) and changed by a user. Different sets of rules may lead to different results under different circumstances, so it may be advantageous to add, remove, or change rules to fit various circumstances or to run multiple automated decision processes simultaneously (e.g., different automated decision processes for different regions).

The term “configured to” is not intended to mean “configurable to.” An unprogrammed FPGA, for example, would not be considered to be “configured to” perform some specific function, although it may be “configurable to” perform that function and may be “configured to” perform the function after programming.

As used herein, the terms “first,” “second,” etc. are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless specifically stated. For example, references to “first” and “second” automated decision platform would not imply an ordering between the two unless otherwise stated.

As used herein, the term “platform” refers to an environment that includes a set of resources that enables some functionality (for example, in the context of the present disclosure, automated decision making). In some cases, this set of resources may be software resources, such that a platform may be said to be constituted solely of software. In other instances, the set of resources may include software and the hardware on which the software executes. Still further, the resources may constitute specialized hardware that performs the functionality; such specialized hardware may, in some cases, utilize firmware and/or microcode in order to execute. (“Modules” are one type of resource; a given module is operable to perform some portion of the overall functionality of a platform.) The term “platform” is thus a broad term that can be used to refer to a variety of implementations. Unless otherwise stated, use of the term “platform” in this disclosure will be understood to constitute all possible types of implementations unless otherwise stated. Note that a platform need not be capable by itself of performing the specified functionality. Rather, it need only provide the capability of performing the functionality. For example, an automated decision-making platform according to the present disclosure provides resources for performing automated decision making; users may utilize the platform to carry out instances of automated decision making. Embodiments of the automated decision-making platform described herein thus enable the functionality of automated decision making to be performed.

As used herein, a “module” refers to software and/or hardware that is operable to perform a specified set of operations. A module may in some instances refer to a set of software instructions that are executable by a computer system to perform the set of operations. Alternatively, a module may refer to hardware that is configured to perform the set of operations. A hardware module may constitute general-purpose hardware as well as a non-transitory computer-readable medium that stores program instructions, or specialized hardware such as a customized ASIC.

DETAILED DESCRIPTION

Automated decision flows may be employed in a wide variety of computing applications. For example, in the context of electronic payment transactions, one or more automated decision flows may be used in order to determine whether a transaction should be executed or denied. An automated flow may be used to calculate a risk decision for a transaction—e.g. whether a risk of fraud is sufficiently low that the transaction should be allowed. Automated decision flows may also be used for purposes of compliance—e.g., if an automated decision indicates a high probability that a transaction may be a money laundering transaction, or is otherwise prohibited by law or regulation, then the transaction may also be denied. Automated decision flows may also be used to generate recommendations to present to users. In various instances, multiple automated flows can be present for a single task, such as execution of a transaction. The inventors have recognized various deficiencies in previous implementations of automated decision flows including: (a) when rules or functions that are parts of an automated decision flow are added, removed, or changed the added or changed rules or functions or in some instance the entire ruleset must be compiled, (b) data dependencies of the automated decision flow are not transparent such that orchestration can be used to control access to the data stores in which such data is stored leading to random accesses rather than more efficient controlled accesses, and (c) in various instances applications implementing automated decision flows are designed to operate in a particular computing environment (e.g., running on a particular operating system) and cannot be run in other computing environments. Techniques according to the present disclosure may ameliorate one or more of these shortcomings.

Referring now toFIG.1, a block diagram illustrating an automated decision computer system100in accordance with various embodiments is shown. In various embodiments, automated decision computer system100includes one or more automated decision platforms110, one or more user devices120, one or more orchestrators130, and one or more data stores140. As discussed herein, in various embodiments automated decision platform110receives rules122and/or user-defined functions (UDFs)124and one or more sets of inputs142to determine one or more decisions112. As discussed herein, the various components of automated decision computer system100may be implemented on one or more computer systems (e.g. standalone system, networked system of one or more computing devices, or other configurations). In some of such embodiments, user device120is implemented with a first computer system and automated decision platform110, orchestrator130, and data store140are implemented with a second computer system (e.g., a multi-tenant computer system400shown inFIG.4implemented on a cloud of computer servers).

In various embodiments, automated decision platform110is operable to automate a decision flow to determine one or more decisions112using one or more user-defined rules122and/or UDFs124and based on one or more sets of inputs142. In various embodiments, automated decision platform110includes one or more modeling modules114and one or more automated decision modules116. In various embodiments, modeling module114is operable to generate a graph model (e.g., graph model204inFIG.2) from the rules122and/or UDFs124and to determine, from the graph model, an input dependency model (e.g., input dependency model206inFIG.2) that is indicative of a set of inputs142referred to in the graph model. In various embodiments, the input dependency model is useable by orchestrator130to coordinate accesses to one or more data stores140in which the set of inputs142are stored. In various embodiments, automated decision module116is operable to receive, from data stores140, set of inputs142. Subsequently, automated decision module116is operable to apply inputs142to the graph model to determine one or more automated decisions112. Automated decision platform110is discussed in further detail herein in reference toFIGS.2,3,4, and6. Implementations of the automated decision platform110in accordance with various embodiments are discussed in further detail herein in reference toFIGS.2,3, and4.

In various embodiments, user device120is any of a number of computer systems that a user can access to input rules122and/or UDFs124. User devices120include but are not limited to desktop computers, laptop computers, tablet computers, and mobile phones in various embodiments. In some embodiments, user device120is a terminal used to access a local computer system (e.g., a server or a mainframe computer system) or remote computer system (e.g., a cloud computer system) via a remote access protocol (e.g., a remote desktop, a virtual machine). In various embodiments, user device120communicates with automated decision platform over a local area network, a wide area network (e.g., the Internet), or a combination of both.

In various instances, a user inputs one or more rules122, UDFs124, or a combination of both on user device120. Rules122define actions (e.g., calculations to perform, selections to be made, variables to set, a subsequent rule122or UDF124to apply, etc.) to be performed when one or more conditions are met (e.g., X=TRUE if Y>5, X=FALSE if Y<5). Rules can be far more complex than these simple examples, however, and in some cases may involve dozens or even hundreds of variables that are used in evaluating the rule. Results of the evaluation of a rule may be binary (e.g. yes/no) in some cases but may also produce one of a set of defined multiple outcomes (e.g. exclusively one of A, B, C, or D) and/or may produce a numeric value which may be within a particular bounded range (e.g. 0.00 to 100.00). Evaluation of a rule may in some cases also produce multiple types of outputs (e.g. risk score 96.72, deny transaction).

In some instances, a particular rule122takes as input (a) information stored in data store140(e.g., input142) and/or (b) information output by another rule122and/or UDF124. UDFs124define one or more actions to perform in association with one or more rules122(e.g., writing to a log file, applying a machine-learning algorithm to an input and/or output of a rule122, converting a datatype of a variable from one datatype to a different datatype). Accordingly, in some instances, a particular UDF124take as input (a) information stored in data store140(e.g., input142, which may represent multiple values) and/or (b) information output by a rule122and/or another UDF124. Collectively, rules122(and in various embodiments rules122and UDFs124) define an automated decision flow that takes input142and determines one or more decisions112by applying the input142to the decision flow. Rules122, UDFs124, and decisions112are discussed in more detail in reference toFIGS.2,3,5, and6.

Orchestrator130controls accesses to data stores140to improve the efficiency of such accesses compared to random accesses. In various embodiments, orchestrator130is implemented by software running on a computer system (e.g., a desktop computer, a laptop computer, a tablet computer, a mobile phone, a server) or a plurality of computer systems (e.g., a network of servers implementing a cloud computing platform). In other embodiments, orchestrator130is implemented in specialized hardware (e.g., on an FPGA) or in a combination of hardware and software. In various instances, orchestrator130is operable to determine one or more of: an order in which to perform the accesses, which of the accesses can be performed in parallel (and which must be performed in serial), or how long the accesses will take to perform and is operable to cache accessed information to speed up repeated queries. As discussed in further detail in reference toFIG.2, in various embodiments, orchestrator130is operable to make such determinations using an input dependency model (e.g., input dependency model206inFIG.2). In various instances, orchestrator130is thus operable to increase the efficiency of access to data stores140.

In various embodiments, the one or more data stores140are configured to store information for subsequent retrieval. As discussed herein, various rules122and/or UDFs124call for information stored in the data stores140in various instances. For example, a rule122might refer to a record of previous transactions as input and output a decision based on the record of previous transactions (e.g., a fraud score, a recommendation for a product or service, a classification for a subsequent transaction). Data stores140may also include a variety of other information, including information specific to a user computing device (e.g. hardware/software configuration information like operating system version, web browser version, screen size, etc.), network information such as IP address etc., transaction information such as destination shipping address, and user account information such as country of citizenship, home address, etc. In various embodiments, the information stored in data stores140that is referred to by one or more rules122and/or UDFs124is accessed according to the control of orchestrator130, and is sent to automated decision platform110as part of the set of inputs142.

As discussed herein, automated decision platform110is useable to improve some or all of the deficiencies in prior automated decision flows identified above. As discussed in further detail in reference toFIGS.2and5, in various embodiments because rules122and many UDFs124(although more complex UDFs124may need to be complied as discussed herein) can be managed released as configurations represented with structured text (e.g., XML, JSON, YAML), such rules122and UDFs124can be used by automated decision platform110without having to compile the rules122and/or UDFs124. As used herein, “compile” includes a process by which a high-level programming language is converted to a lower-level executable and/or interpretable language (e.g. turning Java™ source code into Java™ byte code). Because compilation is not performed on various rules122and UDFs124in various embodiments, the time needed to change the graph model used to implement the rules122and UDFs124is reduced, allowing for an ability to add, change, or remove rules122and UDFs124faster compared to techniques in which rules must be compiled (e.g., an automated decision platform running on rules written in Java™). This ability to use rules without compiling them into a rule engine can be particularly advantageous when dealing with a rule engine that may execute rule sets from multiple sources, as changes can be made to one or more rules without having to re-compile a large set of rules used by the rule engine.

Additionally, in various embodiments, modeling module114is operable generate an input dependency model that is useable by orchestrator130to control accesses to data stores140. Such controlled access allows the data stores140to be accessed in a more time and/or computationally efficient way compared to random accesses by, for example, performing accesses in parallel, sequencing accesses to reduce loading times, etc. Moreover, because the automated decision platform110itself may be flexibly implemented on various operating systems, multiple instances of automated decision platform110may be implemented on a single computer system (e.g., the multi-tenant computer system400inFIG.4) and/or on user devices120(e.g., as shown inFIGS.3and4). Accordingly, in addition to allowing for faster changes to the automated decision flow by adding, changing, or removing rules122and/or UDFs124, the automated decision flows that are implemented using automated decision platform110may be implemented faster, with fewer computational resources, and with more flexibility compared to previous methods.

Referring now toFIG.2, an expanded block diagram is shown of automated decision platform110in accordance with various embodiments. In various embodiments, automated decision platform110is implemented as a number of services including a decision service200which implements modeling module114and automated decision model116, a gateway service210, an analytics service220, and a data access service230. In various embodiments, these various services are made available (e.g., through application programming interfaces) to other applications running on the same computer system as automated decision platform110and, in embodiments, to applications running on other computer systems (e.g., over a LAN or WAN connection).

Decision service200includes modeling module114and automated decision module116. In various embodiments, modeling module114includes one or more parser modules202and a rule testing module208. In various embodiments, the one or more parser modules202are operable to (a) generate graph model204from rules122and/or UDFs124and (b) analyze graph model204to generate input dependency model206. In various embodiments, separate parser modules202are used such that a first parser module202is operable to generate the graph model204and a second parser module202is used to generate input dependency model206from graph model204.

As discussed in further detail in reference toFIG.5, in various embodiments, rules122and/or UDFs124are stored as structured text that uses syntax such as punctuation and/or spacing to indicate rule packages and individual rules and defined commands and syntax to represent rules in a manner similar to a high-level programming language. Parser module202is operable to generate graph model204from this structured text by: (a) determining where the various rules122and/or UDFs124begin and end in the structured text; (b) determining a sequence in which the rules122and/or UDFs124should be applied, (c) identifying one or more inputs, one or more conditions, and one or more actions defined by the various rules122, (d) identifying one or more inputs and one or more actions defined by the various UDFs124, and (e) determining for each given rule122and/or UDF124the other rules122and/or UDFs124that supply input or receive output from that given rule122and/or UDF124. In various embodiments, parser module202(or a second parser module202) is operable to analyze graph model204(or determine directly from rules122and/or UDFs124) what input142is referred to by the graph model204and might be used to determine decisions112. In such embodiments, parser module202generates input dependency model206.

In various embodiments, graph model204is any of a number of graph models in which rules122and UDFs124are represented as nodes and connections between rules122and UDFs124are represented as edges. For example, if a rule122calls a UDF124, then this network may be represented by a first node representing rule122, a second node representing UDF124, and an edge between the two nodes representing the information flow between the rule122and UDF124. In a second example, if a UDF124converts the datatype of information stored in a data store140and then outputs the information in a converted datatype to a rule122, this network may be represented with a third node representing the UDF124, a fourth node representing rule122, and an edge between the two nodes representing the information flow between the UDF124and rule122. In various embodiments, the graph model204generated by parser module202is a directed acyclic graph model.

In various embodiments, parser module202generates input dependency model206based on graph model204. Alternatively or additionally, parser module202generates input dependency model206based on the rules122and/or UDFs124used to generate graph model204. In various embodiments, input dependency model206indicates, for individual inputs142of the set of inputs142: a name of the individual input, a namespace of the individual input, a location of the individual input, and a datatype of the individual input. In various embodiments, some more complex UDFs124(e.g., machine learning model inferencing, existing machine learning feature processing algorithms, interacting with existing APIs) can be written in a general-purpose programming language (e.g., Java™) that needs compilation. In such embodiments, these complex UDFs124are compiled and represented in graph model204and in input dependency model206as a dependency. In such embodiments, during execution of such complex UDFs124a language interoperability function is loaded from data store140and used to run the complex UDFs124.

Using the information contained in input dependency model206, orchestrator130is able to determine which accesses can be performed in parallel, which accesses will take longer than other accesses (e.g., so a longer access can be performed in parallel with two shorter accesses performed in in series such that all three accesses are complete by the time the longer access is complete), in which order the accesses should be completed to account for differences in the amount of time for the access and/or the location of the information stored within data store140(e.g., to group accesses that are located on the same physical media and thereby reduce total access time for the group of accesses), and which separate accesses can be combined (e.g., if access A is loading the last 10 transactions of user1 from transactionHistoryDb and access B is loading the last 20 transactions of user1 from transactionHistoryDb, the orchestrator is able to decide that making only one physical access to db1 (load last 20 transactions) is able to fulfill both requirements). In various embodiments, orchestrator130is operable to cache information for faster repeated accesses compared to accessing the information from data store140.

In various embodiments, modeling module114includes rule testing module208. In such embodiments, rule testing module208is operable to validate rules122and/or UDFs124. In various instances, validating the rules122and/or UDFs124includes determining whether the rules122and/or UDFs124are written in the correct format, are written using the correct syntax and punctuation, and written without improperly using reserved terms. In various instances, validating the rules122and/or UDFs124includes validating dependencies referred to by the rules122and/or UDFs124. In such instances, validating dependences includes determining whether such dependencies (e.g., information referred to by a rule122and/or UDF124) can be located (e.g., have a correct namespace and/or location), have a correctly listed datatype, are available for use according to the access privileges of user or user device120.

In various embodiments, automated decision module116is operable to receive input142, apply input142to graph model204, and determine decisions112. In various embodiments, a plurality of nodes of graph model204take as input one or more respective ones of the set of inputs142. In such embodiments, applying the set of inputs142to graph model204includes inputting the respective ones of the set of inputs142to the corresponding nodes. As discussed herein in additional detail in reference toFIG.5, decisions112include one or more actions to be performed as result of applying input142to one or more rules122. In a first example, one or more decisions112include a prediction of whether a particular transaction request is fraudulent (e.g., an action to set a prediction score to a certain amount, another action to compare the prediction score to a fraud predictions threshold score). In a second example, one or more decisions112include determining a recommendation for a product or service (e.g., an action to recommend product A versus product B if a condition is met). In various embodiments, decisions112are output by automated decision platform110by, for example by returning the decisions112to another computer implemented service that called automated decision platform110via an API, displaying decisions112on a user interface to a user, etc.

As discussed herein, in various instances, automated decision platform110is implemented with services other than decision service200. In various embodiments, gateway service210is operable to provide multi-tenant access to the various services of automated decision platform110. As discussed in further detail in reference toFIG.4, in various embodiments, a multi-tenant computer system may implement a plurality of automated decision platforms110as tenants that share the computer resources of the multi-tenant computer system (e.g., to run multiple automated decision platforms110running different sets of rules122and/or UDFs124). In such embodiments, automated decision platforms110may include various services to facilitate implementation as a tenant, generate analytics indicative of performance within a multi-tenant computer system, etc. In various embodiments, analytics service220is operable to provide, in various embodiments, real-time, near-real-time, or batch-based analytics to enable users to evaluate the efficacy of rules122and/or UDFs124to determine whether rules122and/or UDFs124should be added, removed, or modified. In various embodiments, data access service230is operable to facilitate data acquisition, monitoring, processing, access, and governance (e.g., for data stared in data stores140for example).

Referring now toFIG.3, an expanded block diagram of user device120in accordance with various embodiments is shown. In various embodiments, user device120includes a user interface300and an instance of automated decision platform110. In embodiments, an instance of automated decision platform110can be implemented as a script running within a web browser or other application (not shown) on user device120. In other embodiments, automated decision platform110is implemented as an application installed on user device120.

In various embodiments, user interface300is operable to present information to a user and receives information from the user to enable the user to create rules122and/or UDFs124. In various embodiments, user interface300interfaces with automated decision platform110to receive user-defined rules122and/or UDFs124and to present information to the user to enable the user to add, change, or remove rules122and/or UDFs124. In various embodiments, user interface300includes a text box operable to receive text entered by a user (e.g., via a keyboard coupled to user device120) and to display the entered text. As discussed herein, in various embodiments rules122and/or UDFs124include structured text. In various embodiments, user interface300is operable to enable a user to enter such structured text directly. In various embodiments, user interface300includes a graphical user interface that includes, for example, icons, menu choices, etc. representing available input sources, pre-programmed conditions and actions, and links between rules122and/or UDFs124such that a user can select from among these icons to add, change, or remove rules122and/or UDFs124. In such embodiments, user interface300is operable to automatically generate the structured text based on the selections made by the user such that a user is able to add, change, or remove rules122and/or UDFs124without needing a technical understanding of the structured text and its conventions, syntax, punctuation, etc.

In various embodiments, the instance of automated decision platform110running on user device120(and/or elsewhere) includes a modeling module114as discussed herein. Modeling module114includes a mockup data store310that is useable by rule testing module208to verify the dependencies of rules122and/or UDFs124added or changed by the user. In various embodiments, mockup data store310includes information that is a sample of or is representative of the information stored in data stores140that is useable by rule resting module208to verify dependencies. In some embodiments, mockup data store310includes a historical data store of production data that has previously been evaluated using prior rules122and/or UDFs, which allows for evaluation of new or changed rules122and/or UDFs124in view of the performance under the prior rules122and/or UDFs. Such dependencies may be verified, for example, by determining that the various rules122and/or UDFs124properly invoke information stored in mockup data store310(e.g., by calling the correct namespace and/or location, by identifying the correct datatype, etc.). In various instances, mockup data store310also stores information that is useable by rule testing module208to simulate the performance of a set of rules122and/or UDFs124(e.g., to evaluate a simulated catch rate or false positive rate of a fraud detection ruleset, to evaluate the projected customer response to an advertising campaign that recommends products to customers according to a product recommendation ruleset).

Because an instance of automated decision platform110and a mockup data store310can be implemented by a user device120, a user is able to input and verify rules122and/or UDFs124without having to connect to an automated decision platform110operating on another computer system (e.g., an automated decision platform110running in a cloud computing environment). Thus, a user is able to input and verify proposed rules122and/or UDFs124at the user's leisure and to test changes before pushing new or changed rulesets to an automated decision platform110running in a production computing environment, for example, while still using the same (or similar) data stores and process flows as the production environment. Moreover, user interface300is able to receive input from users who lack the skills to program in a structured text format, thus enabling rules to be added, changed, or removed by users who are not developers. In contrast, in previous automated decision processes, a rule might need to be written in a programming language and then compiled before being pushed to the production computing environment, in which case only a developer with adequate skill might be able to add or change rules.

Referring now toFIG.4, a block diagram illustrating various embodiments of a plurality of automated decision platforms110implemented on a multi-tenant computer system400and various user devices120is shown. In various embodiments, automated decision platform110is flexible and portable such that an instance of automated decision platform110can be run on various operating systems (e.g., on a first operating system run by multi-tenant computer system400, on a second operating system run by user device120A, on a third operating system run by user device120D). In the embodiment shown inFIG.4, four separate instances of automated decision platform110A,110B,110C,110D and three data stores140B,140C and140D are implemented by multi-tenant computer system400, a first user device120A implements an instance of automated decision platform110A, and a second user device120D implements an instance of automated decision platform110D. As shown inFIG.4, logical separations between tenants are represented by dotted lines and various automated decision platforms110are in communication one of three data stores140A,140B,140C. It will be understood, however, that multi-tenant computer system400may implement any number of instances of automated decision platform110and may be in communication with any number of user devices120that may (or may not) implement separate instances of automated decision platform110and various data stores140in various embodiments.

In various embodiments, multi-tenant computer system400implements a plurality of instances of automated decision platform110, each of which implements an automated decision flow as discussed herein. In various embodiments, some instances of automated decision platform110implement the same decision flow (e.g., for load sharing, for redundancy), but in other instances each implements a different decision flow based on different sets of rules122and/or UDFs124. In various embodiments, some instances of automated decision platform110(e.g.,110A and110B) share access to one or more data store140(e.g.,140B) whereas other instances of automated decision platform110(e.g.,110C and110D) do not share access to data stores (e.g.,140C and140D). In various embodiments, some instances of automated decision platform110may be in production computing environments (e.g., the instance of automated decision platform110is being used to automate decisions for customers) while others are pre-production computing environments (e.g., an instance of automated decision platform110that is used for testing new or modified rules122and/or UDFs124prior to release to the production computing environment).

For example, an instance of pre-production automated decision platform110A implemented within multi-tenant computer system400is shown as being in communication with another instance of automated decision platform110A implemented on user device120A. Automated decision platform110A is implemented in a production computing environment, and is operable to receive information from automated decision platform110A. The instance of automated decision platform110A implemented within multi-tenant computer system400is operable to verify, using data store140B, a second set of rules122and/or UDFs124received from user device120A, and release the second set of rules122and/or UDFs124to automated decision platform110B. In some of such embodiments, automated decision platform110B is running an automated decision flow generated from a first set of rules122and/or UDFs124that were previously verified and released to the production computing environment. In various embodiments, such release is performed by sending the rules122and/or UDFs124themselves, but in other embodiments such release is performed by sending the graph model and/or input dependency model generated from the rules122and/or UDFs124to automated decision platform110B. In various embodiments, neither the first set of rules122and/or UDFs124or the second set of rules122and/or UDFs124are compiled as discussed herein. In various embodiments, releasing the second set of the rules122and/or UDFs124includes replacing the first set of rules122and/or UDFs124in the production computing environment, but in other embodiments the second set of rules122and/or UDFs124is used to supplement or modify the first set of rules122and/or UDFs124. Alternatively or additionally, in some embodiments, some or all of the new or changed rules122and/or UDFs124are verified by the user device120(e.g., user device120A,120D) on which the rules122and/UDFs124were inputted.

Accordingly, a multi-tenant computer system400is operable to implement multiple automated decision platforms110(some or all of which might implement different automated decision flows) for various reasons and to enable changes to be made to the sets of rules122and/or UDFs124defining the automated decision flows. For example, a multi-tenant computer system400might implement various instances of automated decision platforms for various regions to implement sets of rules122and/or UDFs124that account for regional differences (e.g., a first automated decision platform110for fraud detection in the U.S, a second automated decision platform110for fraud detection in Europe, etc.), testing of different sets of rules122and/or UDFs124(e.g., a first automated decision platform110running a first set of rules122and/or UDFs124and a second automated decision platform110running a modified set of rules122and/or UDFs124that are evaluated to see which performed better), or other reasons.

Referring now toFIG.5, pseudocode of an example ruleset500in accordance with various embodiments is shown. As used herein, a “ruleset” refers to a set of rules122and/or UDFs124discussed herein. In various embodiments, a “ruleset” includes only rules122or only UDFs124, but in other embodiments a “ruleset” can include both rules122and UDFs124. The example inFIG.5is non-limiting, and it will be understood that the techniques discussed herein can be applied to various rulesets to accomplish any of a number of classification, fraud prediction, recommendation, or other tasks based on analysis of information stored in various data stores140.

As discussed herein, the rules122and/or UDFs124are stored as structured text. In various embodiments, such structured text is written in a domain-specific language (DSL) that is specific to the context of the automated decision platform(s)110discussed herein. In various instances, because the rules122and/or UDFs124are written in a DSL that is specific to automated decision platform110, these rules122and/or some UDFs124do not need to be compiled to be implemented by automated decision platform110in a decision flow as discussed herein. As discussed herein, in various embodiments, some more complex UDFs124(e.g., machine learning model inferencing, existing machine learning feature processing algorithms, interacting with existing APIs) can be written in a general-purpose programming language (e.g., Java™) that needs compilation.

In various embodiments, the DSL for automated decision platform110provides for various features relating to creation, validation, and execution of rules122and/or UDFs124. In various embodiments, the DSL language (a) is statically typed—in compile time, automated decision platform110can determine from the DSL which field or method of access is allowed or not allowed for certain data type being used; (b) provides for type inference—in compile time, automated decision platform110can automatically identify the data type of some variables, (c) provides null safety—in runtime, the DSL ensures that the field. method access, or operation of a Null value is safe and no NPE should be thrown, (d) provides interoperability with Java™ to support Java function calls (e.g., calls to UDFs124written in Java), (e) provide dependency inference—automated decision platform110can automatically parse the dependencies of a ruleset500, telling what are the inputs and outputs to support dependency validation and generation of the input dependency model206. In various embodiments, the DSL for automated decision platform110provides developers with various features to facilitate the writing and evaluating of rules122and/or UDFs214including but not limited to (a) a debug mode in which all expressions' values (even nested expressions) are printed to logs, (b) an audit model in which the various rules122can be validated to see if the various rules'122conditions can be satisfied, (c) a strict mode in which verification of the rules122and/or UDFs124can be interrupted (or not) if an error is encountered, and (d) a web portal to support rule authoring for the DSL such that developers do not need to know the DSL and can user drag and drop widget which the web portal can translate into DSL. In various embodiments, the DSL for automated decision platform110provides features for (a) simulation of the execution of a ruleset500and (b) monitoring by capturing the rule evaluation result for a ruleset500using one or more historical datasets to which one or more previous rulesets500have been applied (e.g., the performance of a new or changed ruleset500can be evaluated against prior performance using the performance of the one or more previous rulesets500as benchmarks)

In various embodiments, a ruleset500includes one or more variable declaration sections502, one or more rule packages504, and a plurality of individual rules such as first rule510and second rule520. In various embodiments, rules122and/or UDFs124defined in ruleset500are hierarchical such that the one or more rule packages504are at a first level of the hierarchy and each of the one or more rule packages504includes one or more individual rules (e.g., rules510,520) at a second level of the hierarchy. In various embodiments, there is no dependency between individual rules in the same rule package. In such embodiments, because there is no dependency between rules122in the same rule package504, each rule122can be executed in parallel. Moreover, when a user is adding a rule package504or changing an existing rule package504, during validation of the rule package504the user has assurance that each individual rule122will be triggered when the rule package504is triggered in various embodiments.

As shown inFIG.5, a hierarchical ruleset500can access information, evaluate accessed information according to conditions and perform actions based on the evaluation, store intermediate information for use by other rules within ruleset500, and store one or more decisions112. In various embodiments, the one or more variable declaration sections502are used to define variables that are used in ruleset500(e.g., “brand=$taxi_brand:string”). In various embodiments, a definition for a particular variable includes a name of the variable (e.g., “brand”), a namespace or location of information to be accessed from a data store140to be used in the variable (e.g., “$taxi_brand”), and a datatype (e.g., “string”). The various rule packages504in the ruleset500may include rules of the same kind (e.g., rules relating to evaluating dollar amounts of transactions) or purpose (e.g., rules used in a fraud detection algorithm). In various embodiments, rules in a rule package504can pass information to each other via intermediate outputs (line512) that are subsequently accessed (line522). As shown inFIG.5, both rules510and520include one or more if/then statements. Further, as shown inFIG.5, rule520includes various case statements. The final output of ruleset500is determined by rule520in the put command to a location named “result” (e.g., line524). For example, if the stored information indicates that the brand is “Modern Hardware” with a price of 15, then the decision112is “price” and “cheap” in line524.

In addition to the example shown inFIG.5, the following pseudocode provides additional examples of rules122written in DSL.

The following pseudocode is an example of commands that may be used to debug a rule122:

The following pseudocode is an example of commands that may be used to audit a rule122:

The following pseudocode is an example of commands that may be used to rank a list with a rule122:

The following pseudocode is an example of commands that may be used to output a list of results generated by a rule122:

Referring now toFIG.6, a flowchart depicting an automated decision method600is depicted. In the embodiment shown inFIG.6, the various actions associated with method600are implemented by automated decision platform110. In various embodiments, various actions may be added or removed from method600. For example, a graph model may be generated from rules122only with no UDFs124as discussed herein.

At block602, a computer system implementing an automated decision platform110(e.g., multi-tenant computer system400) receives a set of user-defined rules122. As discussed herein, these rules122are useable by a computer service (e.g., a service of automated decision platform110) to automate a decision flow. At block604, the computer system implementing an automated decision platform110receives a set of UDFs124useable by the computer service to supplement the user-defined rules122. At block606, the computer system implementing an automated decision platform110generates a graph model204from the set of user-defined rules122and UDFs124. At block608, the computer system implementing an automated decision platform110determines, from the graph model204, an input dependency model206that is indicative of a set of inputs142referred to in the graph model204. The input dependency model206is useable by an orchestrator130to coordinate accesses to one or more data stores140in which the set of inputs142are stored. At block610, the computer system implementing an automated decision platform110receives, from the one or more data stores140, the set of inputs142. At block612, the computer system implementing an automated decision platform110determines one or more automated decisions112by applying the set of inputs142to the graph model204.

Exemplary Computer System

Turning now toFIG.7, a block diagram of an exemplary computer system700, which may implement the various components of computer system100(e.g., a computer system implementing automated decision platform110, user device120) is depicted. Computer system700includes a processor subsystem780that is coupled to a system memory720and I/O interfaces(s)740via an interconnect760(e.g., a system bus). I/O interface(s)740is coupled to one or more I/O devices750. Computer system700may be any of various types of devices, including, but not limited to, a server system, personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, tablet computer, handheld computer, workstation, network computer, a consumer device such as a mobile phone, music player, or personal data assistant (PDA). Although a single computer system700is shown inFIG.7for convenience, system700may also be implemented as two or more computer systems operating together.

Processor subsystem780may include one or more processors or processing units. In various embodiments of computer system700, multiple instances of processor subsystem780may be coupled to interconnect760. In various embodiments, processor subsystem780(or each processor unit within780) may contain a cache or other form of on-board memory.

System memory720is usable to store program instructions executable by processor subsystem780to cause system700perform various operations described herein. System memory720may be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read only memory (PROM, EEPROM, etc.), and so on. Memory in computer system700is not limited to primary storage such as memory720. Rather, computer system700may also include other forms of storage such as cache memory in processor subsystem780and secondary storage on I/O Devices750(e.g., a hard drive, storage array, etc.). In some embodiments, these other forms of storage may also store program instructions executable by processor subsystem780.

I/O interfaces740may be any of various types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I/O interface740is a bridge chip (e.g., Southbridge) from a front-side to one or more back-side buses. I/O interfaces740may be coupled to one or more I/O devices750via one or more corresponding buses or other interfaces. Examples of I/O devices750include storage devices (hard drive, optical drive, removable flash drive, storage array, SAN, or their associated controller), network interface devices (e.g., to a local or wide-area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system700is coupled to a network via a network interface device750(e.g., configured to communicate over WiFi, Bluetooth, Ethernet, etc.).