Patent ID: 12229223

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG.1illustrates a diagram100of co-creation of an agent110A-D and an environment120A-D. Initially, the agent110A attempts to train in a fully complex environment120A. The system determines that the agent110A is unsuccessfully training and, in response, reduces the complexity. The system then trains the agent to a 33% skill in the simplest environment120B, and then determines to increase the complexity again. The system then trains the agent to a 66% skill in the next complex environment120C, and then determines to increase the complexity again. The system then trains the agent to a 100% skill in the fully complex environment120A, and then determines to stop training as the agent is fully trained.

The environment120is shown in the form of a graph with nodes and connections between the nodes. An environment may refer to a world in which the agent operates. For example, the environment may define the size of a ball that is to be put in a basket, the height of the basket off the ground, the number of players on each team, etc. The environment may include objects, agents, and production rules. For example, objects may include each team, a ball, a basket, and baseboard, agents may include teammates and opponents, and production rules may include descriptions of which agents belong to which teams, that the agents interact with the ball, etc. In the graph, circles may represent objects, squares may represent agents, and arrows may represent production rules. For example, the environment120A may show two teams, each with five agents, where each set of five agents interacts with a hoop and a corresponding baseboard, and all the agents interact with a single ball.

As shown, the simplest environment120B may include a single agent trying to move a basketball within one foot of the agent into a hoop, where the hoop is zero feet off the ground and has a diameter of fifty feet with a baseboard that is five feet high and ten feet wide. The next complex environment120C may include the agent with four other agents on a same team, where the ball starts more than six feet away from all the agents, the hoop is ten feet off the ground and has a diameter of eighteen inches with a baseboard that is 3.5 feet high and six feet wide. The fully complex environment120A may include the agent with four other agents on a same team and five other agents on another team, where the ball starts more than six feet away from all the agents, the hoop is ten feet off the ground and has a diameter of eighteen inches with a baseboard that is 3.5 feet high and six feet wide.

FIG.2illustrates a block diagram of an example system200for agent environment co-creation. The system200includes an environment co-creator210and an agent trainer220. The environment co-creator210may receive a variable complexity environment and an untrained agent (A), interact with the agent trainer220to train the agent (B-H), and then provide the trained agent (I).

For example, the environment co-creator210may receive a definition for an environment that specifies that an agent is to throw a basketball into a basketball hoop where the number of players, number of teams, size of the basket, height of the basket, size of the baseboard, and starting distance of the ball varies based on complexity, then the co-creator210may interact with the agent trainer220to train an untrained agent to make a basket in the environment at the environment's highest complexity, and then provide the trained agent for use.

The definition for the environment may define various production rules. For example, a production rule may be that the height of the hoop is ten feet off the ground and another production rule may be that the hoop is zero feet off the ground. In another example, a production rule may be that the number of teammates is one and another production rule may be that the number of teammates is four. The definition for the environment may define how each production rule varies based on complexity for the environment. For example, the definition may define that the height of the hoop is zero at lowest complexity and ten at highest complexity, and that the number of teammates is zero at lowest complexity and four at highest complexity.

Additionally or alternatively, the definition for the environment may define complexity levels and corresponding production rules for each complexity level. For example, the definition of the environment may define that at a lowest complexity level the hoop is zero feet off the ground and there are zero teammates, at a 2ndlowest complexity level the hoop is one foot off the ground and there are zero teammates, at a 3rdlowest complexity level the hoop is two feet off the ground and there is one teammate, and at a highest complexity level the hoop is ten feet off the ground and there are four teammates.

The environment co-creator210may initially provide an environment with first complexity and an untrained agent to the agent trainer (B). For example, the environment co-creator210may receive a definition of a basketball game environment and the untrained agent, determine a fully complex environment based on the definition of the basketball game, and provide the fully complex environment and the untrained agent to the trainer220.

The environment co-creator210may then receive a semi-trained agent and an indication of a success rate from the agent trainer220(C). For example, the environment co-creator210may receive the semi-trained agent and an indication that the success rate of the semi-trained agent is 20%. The environment co-creator210may then determine whether the success rate satisfies a complexity change criteria (D). For example, the environment co-creator210may determine that the success rate of 20% satisfies a complexity change criteria to decrease the complexity when the success rate is less than 30%, 35%, 40%, or some other amount.

Success may be determined based on a reward function that defines behavior that is rewarded and behavior that is not rewarded. For example, a reward function may define that an agent or an agent's teammate scoring a point is to be rewarded and is considered a success, and losing the ball or missing a throw is not to be rewarded and is considered not a success. The reward function may be specified in the environment provided to the agent trainer220, and may be kept constant as the complexity of the environment increases or decreases.

In response to determining that the complexity change criteria is satisfied, the environment co-creator210may then provide a lower complexity environment and the semi-trained agent to the agent trainer220(E). For example, the environment co-creator210may provide an environment at 50% complexity with the semi-trained agent to the agent trainer220.

The environment co-creator210may then receive a further trained agent and indication of success rate from the agent trainer220in the lower complexity environment (F). For example, the environment co-creator210may receive, from the agent trainer220, a further trained agent and an indication that the success rate is 90% in the lower complexity environment.

The environment co-creator210may then determine whether the success rate satisfies a complexity change criteria (G). For example, the environment co-creator210may determine that the success rate of 90% satisfies a complexity change criteria to increase the complexity when the success rate is greater than or equal to 60%, 70%, 80%, or some other amount.

In response to determining that the complexity change criteria is satisfied, the environment co-creator210may then provide a higher complexity environment and the semi-trained agent to the agent trainer220(H). For example, the environment co-creator210may provide an environment at 100% complexity with the semi-trained agent to the agent trainer220.

The environment co-creator210may then receive a further trained agent and indication of success rate from the agent trainer220in the higher complexity environment (I). For example, the environment co-creator210may receive, from the agent trainer220, a further trained agent and an indication that the success rate is 100% in the 100% complexity environment.

The environment co-creator210may repeat increasing and decreasing complexity of the environment as needed until the success rate of the agent in the 100% complex environment satisfies the completion criteria. For example, the environment co-creator210may determine that a 100% success rate at 100% complexity satisfies a completion threshold of 95% success rate at 100% complexity. In response to determining the completion criteria is satisfied, the environment co-creator210may then provide the trained agent for use (J).

The environment co-creator210may increase or decrease the complexity of the environment according to the definition of the environment. The definition of the environment may specify each production rule, a range of each production rule, and which end of the range is more complex and which is more complex. The environment co-creator210may then decide which production rules to provide to the agent trainer220based on the definition. For example, if lowering a complexity, the environment co-creator210may decrease complexity for each production rule based on using a midpoint between a current value for the production rule and the last value for the production rule from which complexity was increased, and when there were no previous increases in complexity for a production rule, then half the current value and the value for the lowest complexity for the production rule.

In some implementations, the environment co-creator210may increase or decrease the complexity of some or all of the production rules. For example, the environment co-creator210may decrease complexity of only a production rule for hoop size. In another example, the environment co-creator210may decrease complexity of both a production rule for hoop size and a production rule for number of teammates, but leave unchanged a complexity of a production rule for baseboard size.

In some implementations, the environment co-creator may randomly select which production rules to change complexity. For example, the environment co-creator210may determine to decrease complexity be randomly selecting one, two, or some other proper subset of the production rules.

In some implementations, the definition of the environment may specify which production rules to increase or decrease before others, and the environment co-creator210may determine to decrease complexity be increasing or decreasing complexity of the production rules in the order specified. For example, the definition may specify to first reduce complexity in number of opposing player, then number of teammates, then size of baseboard, and then size of hoop, and the environment co-creator may first decrease the number of opposing players, see that the success rate is still too low, then decrease the number of teammates, see that the success rate is still too low, then increase the size of baseboard, etc.

In some implementations, the definition of the environment may specify levels of complexity and corresponding production rules for each level. For example, the definition may specify three levels of complexity, where in the highest level of complexity the production rule for number of opposing teammates is five, the production rule for number of teammates is four, the production rule for the hoop is ten feet off the ground, in the medium level of complexity the production rule for number of opposing teammates is zero, the production rule for number of teammates is four, the production rule for the hoop is ten feet off the ground, and in the lowest level of complexity the production rule for number of opposing teammates is zero, the production rule for number of teammates is zero, the production rule for the hoop is zero feet off the ground. The environment co-creator210may then increase or decrease complexity based on selecting the corresponding level of complexity specified by the definition and using the production rules for the level of complexity.

The agent trainer220may receive an environment and agent from the environment co-creator210, train the agent in the environment to increase a success rate of the agent, and then provide the trained agent and indication of the success rate to the environment co-creator210. In some implementations, the agent may be a neural network and the agent trainer220may be an engine that uses reinforcement learning to train the neural network.

In some implementations, the environment co-creator210may receive initial rewards or a reward function for the agent to be trained. For example, the reward function may specify that putting a basketball into the hoop is to be rewarded.

In some implementations, the functionality of the environment co-creator210and the agent trainer220may be combined. For example, the environment co-creator210may train the agent itself. In the example, the environment co-creator210may determine a success rate of an agent based on a last ten, twenty, fifty or some other number of attempts for the agent, and when the success rate satisfies a decrease complexity threshold, decreases the complexity, when the success rate satisfies an increase complexity threshold, increases a complexity, and when the complexity is at full complexity and the success rate satisfies a completion criteria, stops training and provides the trained agent. In another example, the success rate of the agent may be determined from all attempts for the agent for a particular environment complexity where attempts stop once the agent no longer increases in success rate after twenty, fifty, or some other number of attempts.

FIG.3illustrates a flowchart of an example process300for agent environment co-creation. The process300may be performed by the system200or some other system.

The process300includes determining a success rate of an agent in an environment with a first complexity (310). For example, the environment co-creator210may determine a success rate of 20% for an agent in an environment with 100% complexity. In some implementations, determining a success rate of the agent in an environment with a first complexity includes receiving, from the agent trainer, the success rate. For example, the environment co-creator210may receive an indication from the agent trainer220that the success rate is 90% for an agent in an environment at 50% complexity.

The process300includes determining that the success rate satisfies a complexity change criteria (320). For example, the environment co-creator210may determine the success rate of 20% satisfies a complexity change criteria. In some implementations, determining that the success rate satisfies a complexity change criteria includes determining that the success rate is greater than or equal to a threshold rate. For example, the environment co-creator210may determine that the success rate of 90% satisfies a complexity change criteria of a success rate of 70% or greater resulting in an increase in complexity.

The process300includes determining a second complexity that has a greater complexity than the first complexity (330). For example, the environment co-creator210may determine the second complexity to have greater complexity than the first complexity based on determining a range of eighteen to fifty inches for a diameter of a hoop, that eighteen is most complex and fifty is least complex, and then increase complexity by using a value for the diameter that is closer towards eighteen inches than a current value.

In some implementations, determining the second complexity to have greater complexity than the first complexity includes selecting a next more complex setting from a variable complexity environment. For example, the definition for the environment may specify levels of complexity and corresponding production rules for each level, and the environment co-creator210may determine a level of complexity that the agent was just trained at, determine the next closest level of complexity that is higher complexity, and then determine the second complexity from the productions for the next closest level of complexity.

The process300includes training the agent in the environment with a second complexity (340). For example, the agent may be trained in an environment with 100% complexity after the agent was trained in an environment with 50% complexity. In some implementations, training the agent in the environment with a second complexity includes providing, to an agent trainer, the environment with the second complexity receiving, from the agent trainer, the agent trained in the environment with the second complexity. For example, the environment co-creator210may provide a partially trained agent and the environment with 100% complexity to the agent trainer220and, in response, receive the fully trained agent from the agent trainer220.

The process300includes providing the agent trained in the environment with the second complexity (350). For example, the environment co-creator210may provide the trained agent back to a requestor that provided the variable complexity environment and untrained agent, and the requestor may then deploy the trained agent. In another example, the environment co-creator210may provide the trained agent by deploying the trained agent.

In some implementations, providing the agent trained in the environment with the second complexity includes obtaining a second success rate of the agent in the environment with the second complexity, determining that the second success rate satisfies a completion criteria, and in response to determining that the second success rate satisfies the completion criteria, providing the agent trained in the environment with the second complexity. For example, the environment co-creator210may determine that an agent had a success rate of 100% with an environment at 100% complexity which satisfies a completion criteria of greater than 95% success rate at 100% complexity and, in response, provide the agent for use.

In some implementations, the process300includes determining a second success rate of the agent in an environment with a third complexity, determining that the second success rate satisfies a second complexity change criteria, and in response to determining that the second success rate satisfies the second complexity change criteria, training the agent in the environment with a fourth complexity that is less complex than the third complexity. For example, the environment co-creator210may determine that a success rate of 20% with an environment at 100% complexity, determine that the success rate of 20% satisfies a decrease complexity threshold, and, in response, train the agent in an environment at 50% complexity.

FIG.4illustrates a schematic diagram of an exemplary generic computer system400. The system400includes a processor410, a memory420, a storage device430, and an input/output device440. Each of the components410,420,430, and440are interconnected using a system bus450. The processor410is capable of processing instructions for execution within the system400. In one implementation, the processor410is a single-threaded processor. In another implementation, the processor410is a multi-threaded processor. The processor410is capable of processing instructions stored in the memory420or on the storage device430to display graphical information for a user interface on the input/output device440.

The memory420stores information within the system400. In one implementation, the memory420is a computer-readable medium. In one implementation, the memory420is a volatile memory unit. In another implementation, the memory420is a non-volatile memory unit.

The storage device430is capable of providing mass storage for the system400. In one implementation, the storage device430is a computer-readable medium. In various different implementations, the storage device430may be a floppy disk device, a hard disk device, a solid state drive, an optical disk device, a tape device, a universal serial bus stick, or some other storage device.

The input/output device440provides input/output operations for the system400. In one implementation, the input/output device440includes a keyboard and/or pointing device. In another implementation, the input/output device440includes a display unit for displaying graphical user interfaces.

The features described can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by a programmable processor; and method steps that can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input to the computer.

The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network. Examples of communication networks include, e.g., a LAN, a WAN, and the computers and networks forming the Internet.

The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a network, such as the described one. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers may be part of a cloud, which may include ephemeral aspects.