Planner for an objective-effectuator

In some implementations, a method includes obtaining an objective for a computer-generated reality (CGR) representation of an objective-effectuator. In some implementations, the objective is associated with a plurality of time frames. In some implementations, the method includes determining a plurality of candidate plans that satisfy the objective. In some implementations, the method includes selecting a first candidate plan of the plurality of candidate plans based on a selection criterion. In some implementations, the method includes effectuating the first candidate plan in order to satisfy the objective. In some implementations, the first candidate plan triggers the CGR representation of the objective-effectuator to perform a series of actions over the plurality of time frames associated with the objective.

TECHNICAL FIELD

The present disclosure generally relates to a planner for an objective-effectuator.

BACKGROUND

Some devices are capable of generating and presenting computer-generated reality (CGR) environments. Some CGR environments include virtual environments that are simulated replacements of physical environments. Some CGR environments include augmented environments that are modified versions of physical environments. Some devices that present CGR environments include mobile communication devices such as smartphones, head-mountable displays (HMDs), eyeglasses, heads-up displays (HUDs), and optical projection systems. Most previously available devices that present CGR environments are ineffective at presenting representations of certain objects. For example, some previously available devices that present CGR environments are unsuitable for presenting representations of objects that are associated with an action.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods for determining a plan to satisfy an objective of an objective-effectuator. In various implementations, a device includes a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, a method includes obtaining an objective for a computer-generated reality (CGR) representation of an objective-effectuator. In some implementations, the objective is associated with a plurality of time frames. In some implementations, the method includes determining a plurality of candidate plans that satisfy the objective. In some implementations, the method includes selecting a first candidate plan of the plurality of candidate plans based on a selection criterion. In some implementations, the method includes effectuating the first candidate plan in order to satisfy the objective. In some implementations, the first candidate plan triggers the CGR representation of the objective-effectuator to perform a series of actions over the plurality of time frames associated with the objective.

In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs. In some implementations, the one or more programs are stored in the non-transitory memory and are executed by the one or more processors. In some implementations, the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions that, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.

DESCRIPTION

Examples of CGR include virtual reality and mixed reality.

The present disclosure provides methods, systems, and/or devices for generating a plan that satisfies an objective of an objective-effectuator. An objective-effectuator performs a set of one or more actions in order to effectuate an objective of the objective-effectuator. Some objectives span multiple time frames. For example, some objectives span several minutes, hours or days. A planner generates a plan for satisfying an objective that satisfies multiple time frames. The planner identifies various candidate plans that satisfy the objective. Each candidate plan is associated with a corresponding confidence score. The planner selects one of the candidate plans based on the respective confidence scores. The objective-effectuator effectuates the selected plan in order to satisfy the objective of the objective-effectuator. The selected plan triggers a computer-generated reality (CGR) representation of the objective-effectuator to perform a series of actions over the time frames associated with the objective. For example, if the objective spans twenty minutes, then the selected plan triggers the CGR representation of the objective-effectuator to perform various actions over the twenty minutes in order to satisfy the objective.

FIG. 1is a block diagram of an example operating environment100in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment100includes a controller102and an electronic device103. In the example ofFIG. 1, the electronic device103is being held by a user10. In some implementations, the electronic device103includes a smartphone, a tablet, a laptop, or the like.

As illustrated inFIG. 1, the electronic device103presents a computer-generated reality (CGR) environment106. In some implementations, the CGR environment106is generated by the controller102and/or the electronic device103. In some implementations, the CGR environment106includes a virtual environment that is a simulated replacement of a physical environment. In other words, in some implementations, the CGR environment106is synthesized by the controller102and/or the electronic device103. In such implementations, the CGR environment106is different from the physical environment where the electronic device103is located. In some implementations, the CGR environment106includes an augmented environment that is a modified version of a physical environment. For example, in some implementations, the controller102and/or the electronic device103modify (e.g., augment) the physical environment where the electronic device103is located in order to generate the CGR environment106. In some implementations, the controller102and/or the electronic device103generate the CGR environment106by simulating a replica of the physical environment where the electronic device103is located. In some implementations, the controller102and/or the electronic device103generate the CGR environment106by removing and/or adding items from the simulated replica of the physical environment where the electronic device103is located.

In some implementations, the CGR environment106includes various CGR representations of objective-effectuators, such as a boy action figure representation108athat represents a boy objective-effectuator, a girl action figure representation108bthat represents a girl objective-effectuator, a robot representation108cthat represents a robot objective-effectuator, and a drone representation108dthat represents a drone objective-effectuator. In some implementations, the objective-effectuators represent (e.g., model behavior of) characters from fictional materials, such as movies, video games, comics, and novels. For example, the boy action figure representation108arepresents (e.g., models behavior of) a ‘boy action figure’ character from a fictional comic, and the girl action figure representation108brepresents (e.g., models behavior of) a ‘girl action figure’ character from a fictional video game. In some implementations, the CGR environment106includes objective-effectuators that represent (e.g., model behavior of) respective characters from different fictional materials (e.g., from different movies/games/comics/novels). In various implementations, the objective-effectuators model behavior of physical entities (e.g., tangible objects). For example, in some implementations, the objective-effectuators represent equipment (e.g., machinery such as planes, tanks, robots, cars, etc.). In the example ofFIG. 1, the robot representation108crepresents a robot object-effectuator that models the behavior of a robot, and the drone representation108drepresents a drone objective-effectuator that models the behavior of a drone. In some implementations, the objective-effectuators represent (e.g., model behavior of) fictional entities (e.g., characters, equipment, etc.) from fictional materials. In some implementations, the objective-effectuators represent (e.g., model behavior of) entities from a physical environment, including entities located inside and/or outside of the CGR environment106. In various implementations, an objective-effectuator models the behavior of an entity by manipulating a CGR representation of the objective-effectuator in order to provide an appearance that the CGR representation of the objective-effectuator is performing a set of one or more actions that are within a similarity threshold of actions that the entity performs.

In various implementations, the objective-effectuators perform one or more actions in order to effectuate (e.g., complete/satisfy/achieve) one or more objectives. In some implementations, the objective-effectuators perform a sequence of actions. In some implementations, the controller102and/or the electronic device103determine the actions that the objective-effectuators perform. In some implementations, the actions of the objective-effectuators are within a degree of similarity to (e.g., within a similarity threshold of) actions that the corresponding entities (e.g., characters or objects) perform in the fictional material. In the example ofFIG. 1, the girl action figure representation108bis performing the action of flying (e.g., because the corresponding ‘girl action figure’ character is capable of flying, and/or the ‘girl action figure’ character frequently flies in the fictional materials). In the example ofFIG. 1, the drone representation108dis performing the action of hovering (e.g., because drones in physical environments are capable of hovering). In some implementations, the controller102and/or the electronic device103obtain the actions for the objective-effectuators. For example, in some implementations, the controller102and/or the electronic device103receive the actions for the objective-effectuators from a remote server that determines (e.g., selects) the actions.

In various implementations, an objective-effectuator performs an action in order to satisfy (e.g., complete or achieve) an objective. In some implementations, an objective-effectuator is associated with a particular objective, and the objective-effectuator performs actions that improve the likelihood of satisfying that particular objective. In some implementations, CGR representations of the objective-effectuators are referred to as CGR objects. In some implementations, an objective-effectuator representing (e.g., modeling the behavior of) a character is referred to as a character objective-effectuator. In some implementations, a character objective-effectuator performs actions to effectuate a character objective. In some implementations, an objective-effectuator representing (e.g., modeling the behavior of) an equipment is referred to as an equipment objective-effectuator. In some implementations, an equipment objective-effectuator performs actions to effectuate an equipment objective. In some implementations, an objective effectuator representing (e.g., modeling the behavior of) an environment is referred to as an environmental objective-effectuator. In some implementations, an environmental objective-effectuator performs environmental actions to effectuate an environmental objective.

In some implementations, the CGR environment106is generated based on a user input from the user10. For example, in some implementations, the electronic device103receives a user input indicating a terrain for the CGR environment106. In such implementations, the controller102and/or the electronic device103configure the CGR environment106such that the CGR environment106includes the terrain indicated via the user input. In some implementations, the user input indicates environmental conditions for the CGR environment106. In such implementations, the controller102and/or the electronic device103configure the CGR environment106to have the environmental conditions indicated by the user input. In some implementations, the environmental conditions include one or more of temperature, humidity, pressure, visibility, ambient light level, ambient sound level, time of day (e.g., morning, afternoon, evening, or night), and precipitation (e.g., overcast, rain, or snow). In some implementations, the user input specifies a time period for the CGR environment106. In such implementations, the controller102and/or the electronic device103maintain and present the CGR environment106during the specified time period.

In some implementations, the controller102and/or the electronic device103determine (e.g., generate) actions for the objective-effectuators based on a user input from the user10. For example, in some implementations, the electronic device103receives a user input indicating placement of the CGR representations of the objective-effectuators. In such implementations, the controller102and/or the electronic device103position the CGR representations of the objective-effectuators in accordance with the placement indicated by the user input. In some implementations, the user input indicates specific actions that the objective-effectuators are permitted to perform. In such implementations, the controller102and/or the electronic device103select the actions for the objective-effectuators from the specific actions indicated by the user input. In some implementations, the controller102and/or the electronic device103forgo actions that are not among the specific actions indicated by the user input.

In various implementations, the controller102and/or the electronic device103generate a plan to satisfy an objective of an objective-effectuator. In some implementations, the plan triggers the CGR representation of the objective-effectuator to perform a series of actions over a set of time frames associated with the objective. In some implementations, the plan specifies types of actions for the CGR representation of the objective-effectuator to perform in order to advance (e.g., satisfy) the objective of the objective-effectuator. In some implementations, the plan includes specific actions for the CGR representation of the objective-effectuator to perform in order to advance the objective of the objective-effectuator.

In some implementations, the plan includes a timeline which orders the actions for the CGR representation of the objective-effectuator over the set of time frames associated with the objective. In some implementations, the plan distributes the actions for the CGR representation of the objective-effectuator across the set of time frames associated with the objective. For example, in some implementations, the plan distributes the actions across the set of time frames such that the actions are spread approximately evenly across the set of time frames. In some implementations, the plan includes performing at least one action during each of the time frame in the set of time frames.

In some implementations, the controller102and/or the electronic device103generate the plan for an objective-effectuator based on a plot template associated with the CGR environment106. In some implementations, the plot template defines a type of plot for the CGR environment106. Example plot templates include a mystery plot template, a disaster plot template, a comedy plot template and a rescue plot template. In some implementations, the plan triggers actions that are consistent with the plot template for the CGR environment106. For example, if the plot template for the CGR environment106is a comedy plot template, then the plan for an objective-effectuator triggers actions that satisfy a comedic threshold.

In some implementations, the user10wears a head-mountable device (HMD). In various implementations, the HMD operates in substantially the same manner as the electronic device103shown inFIG. 1. In some implementations, the HMD performs substantially the same operations as the electronic device103shown inFIG. 1. For example, in some implementations, the HMD, being worn by the user10, presents (e.g., displays) the CGR environment106according to various implementations. In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays the CGR environment106. In some implementations, the HMD includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. For example, in some implementations, the electronic device103can be attached to the head-mountable enclosure. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., the electronic device103). For example, in some implementations, the electronic device103slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) the CGR environment106.

FIG. 2is a block diagram of an example system200that generates plans for various objective-effectuators in a CGR environment. To that end, the system includes planners204a. . .204e, objective-effectuator engines, and an emergent content engine250. In various implementations, the emergent content engine250provides objectives254for various objective-effectuators. The planners204a. . .204egenerate plans206a. . .206efor the objective-effectuators based on the objectives254. The planners204a. . .204eprovide the plans206a. . .206eto the objective-effectuator engines. The objective-effectuator engines generate actions210a. . .210ebased on the plans206a. . .206e. The actions210a. . .210eare performed by CGR representations of the objective-effectuators.

In some implementations, a planner generates a plan that triggers a CGR representation of a particular objective-effectuator to perform actions in accordance with the plan. In the example ofFIGS. 1 and 2, the planner204agenerates the plan206awhich triggers the boy action figure representation108ato perform the actions210a. The planner204bgenerates the plan206bwhich triggers the girl action figure representation108bto perform the actions210b. The planner204cgenerates the plan206cwhich triggers the robot representation108cto perform the actions210c. The planner204dgenerates the plan206dwhich triggers the drone representation108dto perform the actions210d. The planner204egenerates the plan206ewhich triggers an environment of the CGR environment106to perform the actions210e.

In various implementations, the planners204a. . .204eselect the plans206a. . .206efrom a set of possible plans206stored in a datastore. In some implementations, the possible plans206are obtained from corresponding fictional source material. For example, in some implementations, the controller102and/or the electronic device103shown inFIG. 1perform semantic analysis on actions performed in the fictional source material and determine (e.g., construct) the possible plans206based on the actions. In some implementations, the possible plans206vary for different objective-effectuators. For example, in some implementations, the possible plans206for the girl action figure representation108binclude planning a rescue operation and planning a fight, whereas the possible plans206for the boy action figure representation108ainclude planning a comedy routine and planning a social gathering.

In some implementations, the planners204a. . .204egenerate the plans206a. . .206ebased on a set of possible actions209stored in a datastore. In some implementations, the set of possible actions209represent actions that the CGR representations of objective-effectuators are capable of performing in a CGR environment. For example, the set of possible actions209represent actions that the boy action figure representation108a, the girl action figure representation108b, the robot representation108cand/or the drone representation108dare capable of performing. In some implementations, the planners204a. . .204egenerate the plans206a. . .206esuch that the plans206a. . .206ecan be satisfied (e.g., carried out) with the set of possible actions209.

In the example ofFIG. 2, the system200includes objective-effectuator engines that generate actions in accordance with corresponding plans. For example, a character engine208agenerates the actions210afor the boy action figure representation108ain accordance with the plan206a. A character engine208bgenerates the actions210bfor the girl action figure representation108bin accordance with the plan206b. An equipment engine208cgenerates the actions210cfor the robot representation108c(e.g., responses for the robot representation108c) in accordance with the plan206c. An equipment engine208dgenerates the actions210dfor the drone representation108d(e.g., responses for the drone representation108d) in accordance with the plan206d. An environmental engine208egenerates environmental responses210efor an environment of the CGR environment106based on the plan206e.

In various implementations, the planners204a. . .204egenerate the plans206a. . .206ebased on a set of parameters211. In some implementations, the parameters211include a set of initialization conditions for a CGR environment. In some implementations, the parameters211are provided by an operator (e.g., a human operator, for example, a user of a device). In some implementations, the parameters211are generated by a CGR environment. In some implementations, the parameters211are provided by objective-effectuators that are instantiated in the CGR environment. In some implementations, the parameters211include initial/end states212for the CGR environment106. In some implementations, the parameters211include environmental conditions214for the CGR environment106.

In some implementations, the planners204a. . .204egenerate the plans206a. . .206ebased on the initial/end states212for the CGR environment106. In some implementations, the CGR environment106is associated with a time duration (e.g., a few minutes, hours or days). For example, the CGR environment106is scheduled to last for the time duration associated with the CGR environment106. In some implementations, the initial/end states212indicate initial/end placements of the characters/equipment at the beginning of the time duration and/or at the end of the time duration associated with the CGR environment106. In such implementations, the plans206a. . .206eare a function of the initial/end placements of the characters/equipment at the beginning/end of the time duration associated with the CGR environment106.

In some implementations, the planners204a. . .204egenerate the plans206a. . .206ebased on the environmental conditions214for the CGR environment106. In some implementations, the environmental conditions214specify weather conditions within the CGR environment106. For example, the environmental conditions214specify an amount of precipitation, an amount of snowfall and/or a visibility level in the CGR environment106. In some implementations, the environmental conditions214are specified by the user. In some implementations, the planners204a. . .204eforgo plans that are adversely impacted by the environmental conditions214(e.g., by assigning a low confidence score to adversely-affected plans). For example, if the environmental conditions214indicate that it is raining in the CGR environment106, then a plan to light a camp fire in the CGR environment106is assigned a low confidence score. In various implementations, the environmental conditions214limit the plans206a. . .206egenerated by the planners204a. . .204e.

In some implementations, the emergent content engine250generates the objectives254based on a set of possible objectives252that are stored in a datastore. In some implementations, the possible objectives252are obtained from corresponding fictional source material. For example, in some implementations, the possible objectives252for the girl action figure representation108binclude saving lives, rescuing pets, and/or fighting crime. In some implementations, the planners204a. . .204eutilize the objectives254to generate additional plans and/or to modify previously-generated plans. For example, in some implementations, the planner204autilizes the objectives254to generate an additional plan206aand/or modify a previously-generated plan206afor the boy action figure representation108a.

In various implementations, the character/equipment/environmental engines provide the actions210a. . .210eto a display engine260(e.g., a rendering and display pipeline). In some implementations, the display engine260modifies the CGR representations of the objective-effectuators and/or the environment of the CGR environment106based on the actions210a. . .210e. In various implementations, the display engine260modifies the CGR representations of the objective-effectuators such that the CGR representations of the objective-effectuator can be seen as performing the actions210a. . .210e. For example, if an action for the girl action figure representation108bis to fly, the display engine260moves the girl action figure representation108bwithin the CGR environment106in order to give the appearance that the girl action figure representation108bis flying within the CGR environment106.

FIG. 3Ais a block diagram of an example planner300in accordance with some implementations. In some implementations, the planner300generates a plan314for a corresponding objective-effectuator (e.g., a character objective-effectuator, an equipment objective-effectuator or an environmental objective-effectuator). In some implementations, the plan314triggers a CGR representation of the objective-effectuator (e.g., the boy action figure representation108a, the girl action figure representation108b, the robot representation108c, and/or the drone representation108d) to perform a series of actions in order to satisfy an objective.

In some implementations, different instances of the planner300implement the planners204a. . .204e. For example, a first instance of the planner300implements the planner204awhich generates the plan206afor the boy action figure representation108a. A second instance of the planner300implements the planner204bwhich generates the plans206bfor the girl action figure representation108b. A third instance of the planner300implements the planner204cwhich generates the plan206cfor the robot representation108c. A fourth instance of the planner300implements the planner204dwhich generates the plan206dfor the drone representation108d. A fifth instance of the planner300implements the planner204ewhich generates the plan206efor the environment of the CGR environment106.

In various implementations, the planner300includes a neural network system310(“neural network310”, hereinafter for the sake of brevity), a neural network training system330(“a training module330”, hereinafter for the sake of brevity) that trains (e.g., configures) the neural network310, and a scraper350that provides possible plans360to the neural network310. In various implementations, the neural network310generates the plan314for a corresponding objective-effectuator based on various inputs.

In some implementations, the neural network310includes a long short-term memory (LSTM) recurrent neural network (RNN). In various implementations, the neural network310generates the plan314based on a function of the possible plans360. For example, in some implementations, the neural network310generates the plan314by selecting one of the possible plans360. In some implementations, the neural network310generates the plan314such that the plan314is within a degree of similarity to at least one of the possible plans360.

In some implementations, the neural network310generates the plan314based on the set of possible actions209. As described herein, in some implementations, the set of possible actions209include actions that a CGR representation of an objective-effectuator is capable of performing in a CGR environment. In some implementations, the neural network310generates the plan314such that the plan314can be satisfied (e.g., carried out) based on the set of possible actions209.

In some implementations, the neural network310generates the plan314based on instantiated equipment representations340. In some implementations, the instantiated equipment representations340refer to equipment objective-effectuators that are instantiated in the CGR environment. For example, referring toFIG. 1, the instantiated equipment representations340include the robot representation108cand the drone representation108din the CGR environment106. In some implementations, the plan314includes engaging with one or more of the instantiated equipment representations340. For example, referring toFIG. 1, in some implementations, the plan206bfor the girl action figure representation108bincludes collaborating with the drone representation108dto perform a search and rescue operation.

In some implementations, the neural network310generates the plan314based on instantiated character representations342. In some implementations, the instantiated character representations342refer to character objective-effectuators that are instantiated in the CGR environment. For example, referring toFIG. 1, the instantiated character representations342include the boy action figure representation108aand the girl action figure representation108bin the CGR environment106. In some implementations, the plan314includes engaging with one or more of the instantiated character representations342. For example, referring toFIG. 1, in some implementations, the plan206afor the boy action figure representation108aincludes catching the girl action figure representation108b. In some implementations, the plan206cfor the robot representation108cincludes assisting the boy action figure representation108ain catching the girl action figure representation108b.

In some implementations, the neural network310generates the plan314based on user-specified scene/environment information344. In some implementations, the user-specified scene/environment information344includes the initial/end states212and/or the environmental conditions214shown inFIG. 2. In some implementations, the plan314is a function of the initial/end states212. In some implementations, the initial/end states212indicate initial/end placement for CGR representation of the objective-effectuator, and the plan314includes performing operations that trigger the CGR representation of the objective-effectuator to move away/towards from/to the initial/end placement. In some implementations, the neural network310adjusts the plan314so that the plan314is better suited for the user-specified scene/environment information344. For example, the plan314includes a flying operation for the girl action figure representation108bwhen the user-specified scene/environment information344indicate that the skies within the CGR environment106are clear. In some implementations, the neural network310forgoes the plan314if the plan314is unsuitable for the environment indicated by the user-specified scene/environment information344. For example, if the plan314includes flying the drone representation108d, then the neural network300forgoes the plan314when the user-specified scene/environment information344indicate high winds within the CGR environment106.

In various implementations, the neural network310generates the plan314based on the objectives254from the emergent content engine250. In some implementations, the neural network310generates the plan314in order to satisfy the objective254from the emergent content engine250. In some implementations, the neural network310evaluates the possible plans360with respect to the objectives254. In such implementations, the neural network310generates the plan314by selecting one of the possible plans360that satisfies the objectives254and forgoes selection of the possible plans360that do not satisfy the objectives254.

In various implementations, the neural network310generates the plan314based on one or more characteristic values345associated with the objective-effectuator. In some implementations, the one or more characteristic values345indicate one or more physical characteristics (e.g., structural characteristics) of the CGR representation of the objective-effectuator. For example, the one or more characteristic values345indicate a body material of the CGR representation of the objective-effectuator. In such implementations, the plan314utilizes the physical characteristics that the CGR representation possesses, and does not utilize the physical characteristics that the CGR representation does not possess. For example, if the CGR representation is made from wax, then the plan314avoids going into an area of the CGR environment that is designated as a hot area where there is a risk of melting the wax.

In some implementations, the one or more characteristic values345indicate an accessory that the CGR representation of the objective-effectuator has (e.g., a jet pack for flying). In such implementations, the plan314utilizes the accessories that the CGR representation has, and avoids accessories that the CGR representation does not have. For example, if the CGR representation has the jet pack accessory, then the plan314may include flying. However, if the CGR representation does not have the jet pack accessory, then the plan314may not include flying or the plan314may include taking a CGR airplane to fly.

In some implementations, the one or more characteristic values indicate one or more behavioral characteristics of the CGR representation of the objective-effectuator. In some implementations, the behavioral characteristics include long-term personality traits such as a level of aggressiveness, a level of patience, a level of politeness, etc. In some implementations, the behavioral characteristics include short-term behavioral attributes such as a mood of the CGR representation of the objective-effectuator. In some implementations, the plan314includes actions that rely on behavioral traits which the CGR representation possesses. For example, if the CGR representation has a relatively high level of aggressiveness, then the plan314includes fighting. In some implementations, the plan314forgoes actions that rely on behavioral traits which the CGR representation does not possess. For example, if the CGR representation has a relatively low level of patience, then the plan314does not include waiting.

In some implementations, the neural network310obtains a current state346of the CGR representation of the objective-effectuator, and generates the plan314based on the current state346of the CGR representation of the objective-effectuator. In some implementations, the neural network310obtains the current state346from an emphatic engine that evaluates the current state346of the CGR representation of the objective-effectuator. In some implementations, the emphatic engine determines the current state346based on a body pose of the CGR representation. In some implementations, the emphatic engine determines the current state346based on a mood of the CGR representation of the objective-effectuator. In some implementations, the emphatic engine determines the current state346based on actions that the CGR representation is currently performing and/or has recently performed (e.g., within a threshold time prior to a current time). In some implementations, the emphatic engine determines the current state346based on an objective of the CGR representation of the objective-effectuator.

In some implementations, the neural network310obtains a portion of a knowledge graph347associated with the CGR environment. In some implementations, the knowledge graph347includes state information that represents one or more states (e.g., a current state and/or historical states) of the CGR environment. In some implementations, the portion of the knowledge graph347represents knowledge of the objective-effectuator. In some implementations, the neural network310generates the plan314based on the portion of the knowledge graph347. As such, in some implementations, the plan314is generated based on the knowledge of the objective-effectuator.

In some implementations, a future outcome determiner348determines potential future outcomes349for the plan314, and the neural network310adjusts the plan314based on the potential future outcomes349. In some implementations, the future outcome determiner348determines the potential future outcomes349for various candidate plans (e.g., the candidate plans314a. . .314nshown inFIG. 3B), and the neural network310selects the plan314based on the potential future outcomes349for the candidate plans. In some implementations, the neural network310selects the candidate plan with the most favorable potential future outcome349. In some implementations, the neural network310selects the candidate plan with the potential future outcome349in which the objective254for the objective-effectuator is satisfied. In some implementations, the future outcome determiner348determines the potential future outcomes349by simulating the plan314(e.g., by executing the plan314in a test CGR environment).

In various implementations, the training module330trains the neural network310. In some implementations, the training module330provides neural network (NN) parameters312to the neural network310. In some implementations, the neural network310includes a model of neurons, and the neural network parameters312represent weights for the neurons. In some implementations, the training module330generates (e.g., initializes/initiates) the neural network parameters312, and refines the neural network parameters312based on the plan314generated by the neural network310.

In some implementations, the training module330includes a reward function332that utilizes reinforcement learning to train the neural network310. In some implementations, the reward function332assigns a positive reward to plans that are desirable, and a negative reward to plans that are undesirable. In some implementations, during a training phase, the training module330compares the plans with verification data that includes verified plans. In such implementations, if the plan314is within a degree of similarity to the verified plans, then the training module330stops training the neural network310. However, if the plan314is not within the degree of similarity to the verified plans, then the training module330continues to train the neural network310. In various implementations, the training module330updates the neural network parameters312during/after the training.

In various implementations, the scraper350scrapes content352to identify the possible plans360. In some implementations, the content352includes movies, video games, comics, novels, and fan-created content such as blogs and commentary. In some implementations, the scraper350utilizes various methods, systems, and devices associated with content scraping to scrape the content352. For example, in some implementations, the scraper350utilizes one or more of text pattern matching, HTML (Hyper Text Markup Language) parsing, DOM (Document Object Model) parsing, image processing, and audio analysis in order to scrape the content352and identify the possible plans360. In some implementations, the scraper350extracts actions from the content352and performs semantic analysis on the extracted actions to generate the possible plans360.

In some implementations, an objective-effectuator is associated with a type of representation362, and the neural network310generates the plan314based on the type of representation362associated with the objective-effectuator. In some implementations, the type of representation362indicates physical characteristics of the objective-effectuator, such as characteristics relating to its appearance and/or feel (e.g., color, material type, texture, etc.). In some implementations, the neural network310generates the plan314based on the physical characteristics of the objective-effectuator. In some implementations, the type of representation362indicates behavioral characteristics of the objective-effectuator (e.g., aggressiveness, friendliness, etc.). In some implementations, the neural network310generates the plan314based on the behavioral characteristics of the objective-effectuator. For example, the neural network310generates a fight plan for the boy action figure representation108ain response to the behavioral characteristics including aggressiveness. In some implementations, the type of representation362indicates functional characteristics of the objective-effectuator (e.g., strength, speed, flexibility, etc.). In some implementations, the neural network310generates the plan314based on the functional characteristics of the objective-effectuator. For example, the neural network310generates a flight plan for the girl action figure representation108bin response to the functional characteristics including flying. In some implementations, the type of representation362is determined based on a user input. In some implementations, the type of representation362is determined based on a combination of rules.

In some implementations, the neural network310generates the plan314based on specified plans364. In some implementations, the specified plans364are provided by an entity that controls the fictional materials from where the character/equipment originated. For example, in some implementations, the specified plans364are provided (e.g., conceived of) by a movie producer, a video game creator, a novelist, etc. In some implementations, the possible plans360include the specified plans364. As such, in some implementations, the neural network310generates the plan314by selecting a portion of the specified plans364.

In some implementations, the possible plans360for an objective-effectuator are limited by a limiter370. In some implementations, the limiter370restricts the neural network310from selecting a portion of the possible plans360. In some implementations, the limiter370is controlled by the entity that controls (e.g., owns) the fictional materials from where the character/equipment originated. For example, in some implementations, the limiter370is controlled (e.g., operated and/or managed) by a movie producer, a video game creator, a novelist, etc. In some implementations, the limiter370and the neural network310are controlled/operated by different entities. In some implementations, the limiter370restricts the neural network310from generating plans that breach a criterion defined by the entity that controls the fictional materials.

FIG. 3Bis a block diagram of the neural network310in accordance with some implementations. In the example ofFIG. 3B, the neural network310includes an input layer320, a first hidden layer322, a second hidden layer324, a classification layer326, and a plan selector328. While the neural network310includes two hidden layers as an example, those of ordinary skill in the art will appreciate from the present disclosure that one or more additional hidden layers are also present in various implementations. Adding additional hidden layers adds to the computational complexity and memory demands, but may improve performance for some applications.

In various implementations, the input layer320is coupled (e.g., configured) to receive various inputs. In the example ofFIG. 3B, the input layer320receives inputs indicating the set of possible actions209, the instantiated equipment representations340, the instantiated character representations342, the user-specified scene/environment information344, the objectives254from the emergent content engine250, the one or more characteristic values345, the current state346, and the portion of the knowledge graph347. In some implementations, the neural network310includes a feature extraction module (not shown) that generates a feature stream (e.g., a feature vector) based on the set of possible actions209, the instantiated equipment representations340, the instantiated character representations342, the user-specified scene/environment information344, the objectives254, the one or more characteristic values345, the current state346and/or the portion of the knowledge graph347. In such implementations, the feature extraction module provides the feature stream to the input layer320. As such, in some implementations, the input layer320receives a feature stream that is a function of the set of possible actions209, the instantiated equipment representations340, the instantiated character representations342, the user-specified scene/environment information344, the objectives254, the one or more characteristic values345, the current state346and/or the portion of the knowledge graph347. In various implementations, the input layer320includes a number of LSTM logic units320a, which are also referred to as model(s) of neurons by those of ordinary skill in the art. In some such implementations, an input matrix from the features to the LSTM logic units320aincludes rectangular matrices. The size of this matrix is a function of the number of features included in the feature stream.

In some implementations, the first hidden layer322includes a number of LSTM logic units322a. In some implementations, the number of LSTM logic units322aranges between approximately 10-500. Those of ordinary skill in the art will appreciate that, in such implementations, the number of LSTM logic units per layer is orders of magnitude smaller than previously known approaches (being of the order of O(101)-O(102)), which allows such implementations to be embedded in highly resource-constrained devices. As illustrated in the example ofFIG. 3B, the first hidden layer322receives its inputs from the input layer320.

In some implementations, the second hidden layer324includes a number of LSTM logic units324a. In some implementations, the number of LSTM logic units324ais the same as or similar to the number of LSTM logic units320ain the input layer320or the number of LSTM logic units322ain the first hidden layer322. As illustrated in the example ofFIG. 3B, the second hidden layer324receives its inputs from the first hidden layer322. Additionally or alternatively, in some implementations, the second hidden layer324receives its inputs from the input layer320.

In some implementations, the classification layer326includes a number of LSTM logic units326a. In some implementations, the number of LSTM logic units326ais the same as or similar to the number of LSTM logic units320ain the input layer320, the number of LSTM logic units322ain the first hidden layer322, or the number of LSTM logic units324ain the second hidden layer324. In some implementations, the classification layer326includes an implementation of a multinomial logistic function (e.g., a soft-max function) that produces a number of candidate plans314a. . .314n. In some implementations, the number of candidate plans314a. . .314nis approximately equal to the number of possible plans360. In some implementations, the candidate plans314a. . .314nare associated with corresponding confidence scores316a. . .316nwhich include a probability or a confidence measure that the corresponding plan satisfies the objective254. In some implementations, the outputs do not include plans that have been excluded by operation of the limiter370.

In some implementations, the classification layer326includes the future outcome determiner348. In such implementations, the future outcome determiner348determines potential future outcomes for each of the candidate plans314a. . .314n. In some implementations, the classification layer326determines the confidence scores316a. . .316nfor the candidate plans314a. . .314nbased on the respective potential future outcomes of the candidate plans314a. . .314n. In some implementations, the classification layer326assigns a higher confidence score to candidate plans that result in desirable potential future outcomes, and a lower confidence score to candidate plans that result in undesirable potential future outcomes. In some implementations, a desirable potential future outcome refers to an outcome in which the objective for the objective-effectuator is satisfied, whereas an undesirable potential future outcome refers to an outcome in which the objective for the objective-effectuator is not satisfied.

In some implementations, the plan selector328generates the plan314by selecting the top N candidate plans provided by the classification layer326. For example, in some implementations, the plan selector328selects the candidate plan with the highest confidence score. In some implementations, the top N candidate plans are most likely to satisfy the objectives254. In some implementations, the plan selector328provides the plan314to a rendering and display pipeline (e.g., the display engine260shown inFIG. 2).

FIG. 3Cillustrates a behavior tree (BT)380. In some implementations, the BT380includes a probabilistic behavior tree (PBT). In the example ofFIG. 3C, the BT380generates the plan314based on one or more inputs. As illustrated inFIG. 3C, in some implementations, the inputs to the BT380include indications of the set of possible actions209, the instantiated equipment340, the instantiated characters342, the user-specified scene/environment information344, the objectives254, the characteristic value(s)345, the current state346of the objective-effectuator, the portion of the knowledge graph347, and/or the potential future outcomes349. In some implementations, searching/traversing the BT380with one or more of the inputs shown inFIG. 3Cresults in the plan314.

FIG. 4Ais a flowchart representation of a method400of generating a plan for an objective-effectuator. In various implementations, the method400is performed by a device with a non-transitory memory and one or more processors coupled with the non-transitory memory (e.g., the controller102and/or the electronic device103shown inFIG. 1). In some implementations, the method400is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method400is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

As represented by block410, in various implementations, the method400includes obtaining an objective for a CGR representation of an objective-effectuator. In some implementations, the objective is associated with a plurality of time frames (e.g., several seconds, minutes, hours, days, weeks, months, years or decades). In some implementations, the method400includes receiving the objective from an emergent content engine that generated the objective. For example, as shown inFIG. 2, the planner204areceives the objective254for the boy objective-effectuator, represented by the boy action figure representation108a, from the emergent content engine250. In some implementations, the method400includes generating the objective (e.g., by selecting the objective from a set of possible objectives). In some implementations, the method400includes retrieving the objective from a datastore.

As represented by block420, in various implementations, the method400includes determining a plurality of candidate plans that satisfy the objective. In some implementations, each of the plurality of candidate plans is associated with a corresponding confidence score. For example, as shown inFIG. 3B, the neural network310determines the candidate plans314a. . .314n, and each of the candidate plans314a. . .314nis associated with corresponding confidence scores316a. . .316n.

As represented by block430, in various implementations, the method400includes selecting a first candidate plan of the plurality of candidate plans based on a selection criterion (e.g., based on the respective confidence scores). For example, as shown inFIG. 3B, the plan selector328selects the plan314from the candidate plans314a. . .314nbased on the confidence scores316a. . .316n. In some implementations, the method400includes selecting a candidate plan with a confidence score that satisfies a confidence threshold. For example, selecting a candidate plan with a confidence score that is greater than the confidence threshold. In some implementations, the method400includes selecting the candidate plan with the highest confidence score.

As represented by block440, in various implementations, the method400includes effectuating the first candidate plan in order to satisfy the objective. In some implementations, the first candidate plan triggers the CGR representation of the objective-effectuator to perform a series of actions over the plurality of time frames associated with the objective. In some implementations, the method400includes generation actions in accordance with the plan, and performing the generated actions. For example, as shown inFIG. 2, the planners204a. . .204nprovide the plans206a. . .206nto the objective-effectuator engines in order to allow the objective-effectuator engines to generate the actions210a. . .210nin accordance with the plans206a. . .206n.

Referring toFIG. 4B, as represented by block420a, in some implementations, the method400includes determining the plurality of candidate plans based on one or more characteristic values associated with the objective-effectuator (e.g., the one or more characteristic values345shown inFIGS. 3A-3C). In some implementations, the characteristic values indicate various characteristics of the CGR representation of the objective-effectuator. In such implementations, the method400includes determining the plurality of candidate plans based on the characteristics of the CGR representation of the objective-effectuator. For example, the method400includes assigning higher confidence scores to candidate plans that the characteristics support and assigning lower confidence scores to candidate plans that the characteristics do not support. In some implementations, the one or more characteristic values indicate one or more physical/structural characteristics of the CGR representation of the objective-effectuator (e.g., body material, accessories such as a jet pack, etc.). In some implementations, the one or more characteristic values indicate one or more behavioral characteristics of the CGR representation of the objective-effectuator (e.g., level of aggressiveness, mood, etc.). In some implementations, the one or more characteristic values indicate one or more functional characteristics of the CGR representation of the objective-effectuator (e.g., running, flying, defying gravity, etc.).

As represented by block420b, in some implementations, the method400includes utilizing a behavior tree (BT) (e.g., a probabilistic behavior tree (PBT) to generate the plurality of candidate plans (e.g., the BT380shown inFIG. 3C). In some implementations, the method400includes searching the BT for the plurality of candidate plans based on one or more characteristic values associated with the objective-effectuator. In some implementations, the method400includes traversing the BT for the plurality of candidate plans based on one or more characteristic values associated with the objective-effectuator. For example, in some implementations, the BT generates the candidate plans314a. . .314nshown inFIG. 3B.

As represented by block420c, in some implementations, the method400includes utilizing a reactive neural network to generate the plurality of candidate plans. For example, in some implementations, the reactive neural network generates the candidate plans314a. . .314nshown inFIG. 3B.

As represented by block420d, in some implementations, the method400includes generating the candidate plans based on a current state of the CGR representation of the objective-effectuator (e.g., based on the current state346shown inFIGS. 3A-3C). In some implementations, the method400includes obtaining a current state of the CGR representation of the objective-effectuator, and determining the plurality of candidate plans based on the current state of the CGR representation of the objective-effectuator. In some implementations, the method400includes utilizing an emphatic engine to evaluate the current state of the CGR representation of the objective-effectuator. In some implementations, the method400includes evaluating the current state of the CGR representation of the objective-effectuator based on a body pose of the CGR representation and/or based on actions being performed by the CGR representation. In some implementations, the method400includes providing the current state of the CGR representation of the objective-effectuator as an input to a neural network that generates the plurality of candidate plans (e.g., providing the current state as an input to the neural network310shown inFIG. 3A).

As represented by block420e, in some implementations, the objective-effectuator determines the plurality of candidate plans. In some implementations, the objective-effectuator includes a planner (e.g., the planner300shown inFIG. 3A) that determines the plurality of candidate plans. In some implementations, the planner is integrated into an objective-effectuator engine that generates actions for the CGR representation of the objective-effectuator (e.g., in some implementations, the planner204ais integrated into the character engine208ashown inFIG. 2).

As represented by block420f, in some implementations, the method400includes determining the candidate plans based on known knowledge (e.g., based on the portion of the knowledge graph347shown inFIGS. 3A-3C). In some implementations, each objective-effectuator has access to a limited portion of a knowledge graph, and all objective-effectuators collectively have access to the entire knowledge graph. In such implementations, the method400includes determining the plurality of candidate plans based on the portion of the knowledge graph for which the objective-effectuator has access.

As represented by block420g, in some implementations, the method400includes generating the plurality of candidate plans based on reinforcement learning. For example, in some implementations, the method400includes assigning higher scores to desirable plans during a training phase, and assigning lower scores to undesirable plans during the training phase. In some implementations, the method400includes generating the plurality of candidate plans based on initiation learning.

Referring toFIG. 4C, as represented by block430a, in some implementations, the method400includes determining respective potential future outcomes for each of the plurality of candidate plans (e.g., the potential future outcomes349shown inFIGS. 3A-3C). For example, determining respective potential future outcomes for each of the candidate plans314a. . .314nshown inFIG. 3B.

As represented by block430b, in some implementations, the method400includes determining corresponding confidence scores for the plurality of candidate plans based on the potential future outcomes. For example, determining the confidence scores316a. . .316nfor the candidate plans314a. . .314nshown inFIG. 3B. In some implementations, the method400includes assigning higher confidence scores to candidate plans that result in desirable potential future outcomes, and assigning lower confidence scores to candidate plans that result in undesirable potential future outcomes.

As represented by block430c, in some implementations, the method400includes ranking the plurality of candidate plans based on the respective potential future outcomes. For example, the method400includes assigning higher ranks to candidate plans that result in desirable potential future outcomes, and assigning lower ranks to candidate plans that result in undesirable potential future outcomes. In some implementations, the method400includes ranking the plurality of candidate plans based on the corresponding confidence scores.

As represented by block430d, in some implementations, the first candidate plan limits the series of actions to a permissible set of actions. In some implementations, the first candidate plan limits the series of actions in order to avoid an impermissible set of actions. In some implementations, the first candidate plan guides the objective-effectuator engine in generating the actions. For example, the first candidate plan indicates types of actions for the objective-effectuator engine to generate. In some implementations, the first candidate plan indicates a sequence for different types of actions for the objective-effectuator.

As represented by block430e, in some implementations, the method400includes modifying the BT (e.g., the PBT) based on the selection of the first candidate plan (e.g., modifying the BT380based on the selection of the plan314). In some implementations, the method400includes, after selecting the first candidate plan, assigning more weight (e.g., a higher probability) to a portion of the BT that corresponds to the first candidate plan. In some implementations, the method400includes, after selecting the first candidate plan, assigning less weight (e.g., a lower probability) to a portion of the BT that does not correspond to the first candidate plan.

As represented by block440a, in some implementations, the method400includes displaying the CGR representation of the objective-effectuator performing the series of actions over the plurality of time frames in accordance with the first candidate plan. For example, displaying the boy action figure representation108aperforming the actions210athat were generated in accordance with the plan206ashown inFIG. 2. In some implementations, the method400includes animating (e.g., manipulating) the CGR representation of the objective-effectuator in order to provide an appearance that the CGR representation of the objective-effectuator is performing the series of actions.

FIG. 5is a block diagram of a device500enabled with one or more components of a planner (e.g., one of the planners204a. . .204eshown inFIG. 2, or the planner300shown inFIG. 3A) in accordance with some implementations. While certain specific features are illustrated, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device500includes one or more processing units (CPUs)501, a network interface502, a programming interface503, a memory504, and one or more communication buses505for interconnecting these and various other components.

In some implementations, the network interface502is provided to, among other uses, establish and maintain a metadata tunnel between a cloud hosted network management system and at least one private network including one or more compliant devices. In some implementations, the one or more communication buses505include circuitry that interconnects and controls communications between system components. The memory504includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory504optionally includes one or more storage devices remotely located from the one or more CPUs501. The memory504comprises a non-transitory computer readable storage medium.

In some implementations, the memory504or the non-transitory computer readable storage medium of the memory504stores the following programs, modules and data structures, or a subset thereof including an optional operating system506, a data obtainer510, a candidate plan determiner520, a plan selector530, and a plan effectuator540. In various implementations, the device500performs the method400shown inFIGS. 4A-4C.

In some implementations, the data obtainer510obtains an objective for a CGR representation of an objective-effectuator. In some implementations, the data obtainer510performs the operation(s) represented by block410inFIG. 4A. To that end, the data obtainer510includes instructions510a, and heuristics and metadata510b.

In some implementations, the candidate plan determiner520determines a plurality of candidate plans that satisfy the objective. In some implementations, the candidate plan determiner520performs the operations(s) represented by blocks420and420a. . .420gshown inFIGS. 4A-4B. To that end, the candidate plan determiner520includes instructions520a, and heuristics and metadata520b.

In some implementations, the plan selector530selects a first candidate plan of the plurality of candidate plans based on the respective confidence scores. In some implementations, the plan selector530performs the operations represented by blocks430and430a. . .430eshown inFIGS. 4A and 4C. To that end, the plan selector530includes instructions530a, and heuristics and metadata530b.

In some implementations, the plan effectuator540effectuates the first candidate plan in order to satisfy the objective. In some implementations, the plan effectuator540performs the operations represented by blocks440and440ashown inFIGS. 4A and 4C. To that end, the plan effectuator540includes instructions540a, and heuristics and metadata540b.