Commands for simulation systems and methods

Methods, systems, computer-readable media, and apparatuses for performing, providing, managing, executing, and/or running a distributed simulation are presented. In one or more embodiments, the distributed simulation may comprise a plurality of workers performing the simulation, and workers may send commands to other workers authoritative over entity components. A mapping of entity components to workers may be used to determine a bridge associated with a worker to which to send a command. A request to invoke the command may be transmitted to the worker via the bridge associated with the worker. The worker transmitting the command request may receive a response to the request to invoke the command, such as a success response or a failure response.

FIELD

Aspects described herein generally relate to computers, networking, hardware, and software. More specifically, some aspects described herein relate to a networked system architecture for controlling a distributed and persistent computer-based simulation, including issuing and receiving commands in the simulation.

BACKGROUND

Conventional simulation systems are unable to scale to support very large numbers of objects to simulate those objects in real-time. Such systems have typically relied on a single instance of a simulation engine, running on a single physical or virtual computer system, to simulate the entire simulated world. Consumers of these simulation systems have had to choose between correctness, graphical fidelity, and real-time-interaction, with no solution offering the ability for all three on a large scale system. The magnitude and complexity of the situation is further increased if the consumer desires to simulate complex real-world problems which may require more computing power than a single simulation engine can provide. For example, a simulation of a city may require simulation of a large number of vehicles, pedestrians, bicyclists, traffic patterns, traffic lights, subway systems, transit vehicles, airplanes, and a multitude of other entities that affect and contribute to city life.

In one known approach, computing resources have been statically assigned to a portion of the simulated world. A disadvantage of this approach may be that as the simulated objects, actors, etc. move across the simulated world as the simulation progresses, the simulated objects may congregate on a very small region of the simulated world. If sufficient objects move to the very small region, the computing resources may be overloaded (resulting in slower processing), the simulation may terminate unexpectedly, and/or simulation data may be lost. Another disadvantage of this approach may be that state information of the simulation for a region may be concentrated on a single computing resource and may not be shared or spread across several resources, making fault tolerance or recovery from an unexpected termination difficult and time-consuming. In addition, this approach may not lend itself to easily support stateful migration of simulated objects across region boundaries, and thus simulations usually limit stateful migrations to only players.

These and other problems are addressed herein.

SUMMARY

To overcome limitations in the prior art described above, and to overcome other limitations that will be apparent upon reading and understanding the present specification, aspects described herein describe instantiating, on one or more computing devices, a plurality of workers associated with a computer-based simulation. Each worker of the plurality of workers may be authoritative over one or more entity components of a plurality of entity components associated with the computer-based simulation. Each worker of the plurality of workers may be associated with a bridge of a plurality of bridges. A first worker of the plurality of workers may determine a command for an entity component of the plurality of entity components, where a second worker of the plurality of workers may be authoritative over the entity component. The system may determine, based on a mapping of entity components to workers, a bridge associated with the second worker. The system may transmit, via a bridge associated with the first worker and via the bridge associated with the second worker, a request to invoke the command on the entity component. The first worker may receive, from the second worker and via the bridge associated with the second worker, a response to the request to invoke the command.

In some examples, the first worker of the plurality of workers may determine an identifier for the command for the entity component, and the request to invoke the command may comprise the identifier for the command and a payload for the command. The identifier for the command may comprise a first identifier for the command, and the bridge associated with the second worker may determine a second identifier for the command for the entity component. The second identifier may be different from the first identifier. Additionally or alternatively, receiving the response to the request to invoke the command may comprise receiving the second identifier for the command.

In some examples, after determining the command, the first worker may transmit, to the bridge associated with the first worker, a request to invoke the command on the entity component. The request to invoke the command transmitted to the bridge associated with the first worker may comprise an identifier for the entity component. Determining the bridge associated with the second worker may comprise determining, based on the identifier for the entity component and based on the mapping of entity components to workers, the bridge associated with the second worker. Additionally or alternatively, determining the bridge associated with the second worker may comprise transmitting, by the bridge associated with the first worker and to a resolver associated with the computer-based simulation, a lookup request comprising the identifier for the entity component. Determining the bridge associated with the second worker may also comprise receiving, by the bridge associated with the first worker, from the resolver, and based on transmitting the lookup request, an identifier for the bridge associated with the second worker. Transmitting the request to invoke the command on the entity component may comprise transmitting, by the bridge associated with the first worker and to the bridge associated with the second worker, the request to invoke the command on the entity component.

In some examples, methods described herein may comprise assigning authority over the entity component from the second worker to a third worker of the plurality of workers. The first worker may determine a second command for the entity component. Based on the mapping of entity components to workers, a bridge associated with the third worker may be determined. A request to invoke the second command on the entity component may be transmitted via the bridge associated with the third worker.

In some examples, the response to the request to invoke the command may comprise an indication that invocation of the command on the entity component failed. The response to the request to invoke the command may indicate one or more of a time out for the command, that the command was transmitted during an entity component authority handoff, or that a user-defined failure occurred.

In some examples, a computing device of the one or more computing devices may store, in a database associated with the computer-based simulation, the mapping of entity components to workers. Determining the bridge associated with the second worker may comprise determining, based on the mapping of entity components to workers stored in the database, the bridge associated with the second worker.

Systems and non-transitory computer readable media may be configured to provide and/or support various aspects described herein. These and additional aspects will be appreciated with the benefit of the disclosures discussed in further detail below.

It should be noted that any one or more of the above-described features may be used with any other feature or aspect in isolation or any combination. Features from one embodiment or aspect may be interchanged or used together with one or more features of any other described embodiment or aspect.

DETAILED DESCRIPTION

As a general introduction to the subject matter described in more detail below, aspects described herein are directed towards systems, methods, and techniques for providing a distributed, persistent, and spatially-optimized simulation development environment. Other aspects described herein may allow for the integration of existing non-distributed simulation programs into a large-scale distributed simulation. Yet other aspects described herein may be used to automatically and spatially balance and distribute the simulation workload.

Computer software, hardware, and networks may be utilized in a variety of different system environments, including standalone, networked, virtualized, and/or cloud-based environments, among others.FIG. 1illustrates one example of a block diagram of a spatially-optimized simulation computing device (or system)101in a spatially-optimized simulation computing system100that may be used according to one or more illustrative embodiments of the disclosure. The spatially-optimized simulation computing device101may comprise a processor103for controlling overall operation of the spatially-optimized simulation computing device101and its associated components, including RAM105, ROM107, input/output module109, and memory111. The spatially-optimized simulation computing device101, along with one or more additional computing devices (e.g., network nodes123,125,127,129, and131) may correspond to any one of multiple systems or devices described herein, such as personal mobile devices, client computing devices, proprietary simulation systems, additional external servers and other various devices in a spatially-optimized simulation computing system100. These various computing systems may be configured individually or in combination, as described herein, for providing a spatially-optimized simulation computing system100. In addition to the features described above, the techniques described herein also may be used for allowing integration of existing simulation programs, and for spatially load-balancing the simulation workload across the spatially-optimized simulation computing system100, as will be discussed more fully herein. Those of skill in the art will appreciate that the functionality of spatially-optimized simulation computing device101(or devices123,125,127,129, and131) as described herein may be spread across multiple processing devices, for example, to distribute processing load across multiple computers, to segregate transactions based on processor load, location within a simulated world, user access level, quality of service (QoS), and the like.

The various network nodes123,125,127,129, and131may be interconnected via a network121, such as the Internet. Other networks may also or alternatively be used, including private intranets, corporate networks, local area networks (LAN), wide area networks (WAN), metropolitan area networks (MAN), wireless networks, personal networks (PAN), and the like. Network121is for illustration purposes and may be replaced with fewer or additional computer networks. Network121may have one or more of any known network topology and may use one or more of a variety of different protocols, such as Ethernet. Devices123,125,127,129,131, and other devices (not shown) may be connected to one or more of the networks via twisted pair wires, coaxial cable, fiber optics, radio waves, or other communication media.

It will be appreciated that the network connections shown are illustrative and other means of establishing a communications link between the computers may be used. The existence of any of various network protocols such as TCP/IP, Ethernet, FTP, HTTP and the like, and of various wireless communication technologies such as GSM, CDMA, Wi-Fi, and WiMAX, is presumed, and the various computing devices in spatially-optimized simulation system components described herein may be configured to communicate using any of these network protocols or technologies.

The term “network” as used herein and depicted in the drawings refers not only to systems in which remote computing devices are coupled together via one or more communication paths, but also to stand-alone devices that may be coupled, from time to time, to such systems that have storage capability. Consequently, the term “network” includes not only a “physical network” but also a “content network,” which is comprised of the data which resides across all physical networks.

The Input/Output (I/O) module109may include a microphone, keypad, touch screen, game controller, joystick, and/or stylus through which a user of the spatially-optimized simulation computing device101may provide input, and may also include one or more of a speaker for providing audio output and a video display device for providing textual, audiovisual and/or graphical output. Software may be stored within memory111and/or storage to provide instructions to processor103for enabling a spatially-optimized simulation computing device101to perform various actions. For example, memory111may store software used by a spatially-optimized simulation computing device101, such as an operating system113, application programs115, and an associated internal database117. The database117may include a second database (e.g., as a separate table, report, etc.) That is, the information may be stored in a single database, or separated into different logical, virtual, or physical databases, depending on system design. The various hardware memory units in memory111may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Spatially-optimized simulation computing device101and/or computing devices127,129,131may also be mobile terminals (e.g., mobile phones, smartphones, personal digital assistants (PDAs), notebooks, etc.) including various other components, such as a battery, speaker, and antennas (not shown.)

Aspects described herein may also be operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of other computing systems, environments, and/or configurations that may be suitable for use with aspects described herein include, but are not limited to, personal computers, server computers, hand-held or laptop devices, vehicle-based computing devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network personal computers (PCs), minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

FIG. 2shows a high-level architecture of an illustrative spatially-optimized simulation system. As shown, the spatially-optimized simulation system200may be a single server system, a multi-server system, or a cloud-based system, including at least one virtual server202which may be configured to provide spatially-optimized simulation functionality to the spatially-optimized simulation system200and/or may provide access to the spatially-optimized simulation system200to one or more client computing devices (e.g., computing devices123,125,127,129,131.) A virtual server202may comprise one or more virtual machines240a-240n(generally referred to herein as “virtual machine(s)240”). Each virtual machine240may comprise an instance of a spatial simulation runtime248for instantiating, managing, and monitoring one or more instances of server worker processes249a-249n(generally referred to herein as “worker(s)249.”) As described in further detail below, the spatial simulation runtime248may be configured to automatically spool up or spool down workers249, as needed, based on the instantaneous workload of particular regions of the simulated world generated by the spatially-optimized simulation system.

The one or more instances of the spatial simulation runtime248within a virtual server202may communicate with each other to determine an instance which may serve as a master. For example, the spatial simulation runtime248instances may utilize a consensus protocol to determine a master. A master spatial simulation runtime248instance may be responsible for routing communications between the other spatial simulation runtime248instances within the virtual server202and other spatial simulation runtimes248executing in other virtual servers202. As will be explained in greater detail below, the spatial simulation runtime248may allow for spatially-optimized distributed simulations where simulation workload is automatically distributed across available virtual server(s)202. The virtual server202illustrated inFIG. 2may be deployed as and/or implemented by one or more embodiments of the spatially-optimized simulation computing device101illustrated inFIG. 1or by other known computing devices.

The virtual server202may comprise a hardware layer210with one or more hardware elements that communicate with the virtual server202. Optionally, the hardware layer210may comprise one or more physical disks212, one or more physical devices214, one more physical processors216, and one or more physical memories218. Physical components212,214,216, and218may include, for example, any of the components described above with respect to spatial simulation computing device101. In one example, physical devices214may include a network interface card, a video card, a keyboard, a mouse, an input device, a monitor, a display device, speakers, an optical drive, a storage device, a universal serial bus connection, a printer, a scanner, a network element (e.g., router, firewall, network address translator, load balancer, virtual private network (VPN) gateway, Dynamic Host Configuration Protocol (DHCP) router, etc.), or any device connected to or communicating with virtualization server301. Physical memory218may include any type of memory. In another example, physical memory218may store data, and may store one or more programs, or set of executable instructions. Programs or executable instructions stored in the physical memory218may be executed by the one or more processors216of virtual server202. Virtual server202may further comprise a host operating system220which may be stored in a memory element in the physical memory218and may be executed by one or more of the physical processors216.

Hypervisor230may provide virtual resources to operating systems246a-246nor to workers249executing on virtual machines240in any manner that simulates the operating systems246or workers249having direct access to system resources. System resources may include, but are not limited to, physical disks212, physical devices214, physical processors216, physical memory218, and any other component included in hardware layer210. Hypervisor230may be used to emulate virtual hardware, partition physical hardware, virtualize physical hardware, and/or execute virtual machines that provide computing resources to spatial simulation runtime248and workers249. Hypervisor230may control processor scheduling and memory partitioning for a virtual machine240executing on virtual server202.

Hypervisor230may be Type 2 hypervisor, where the hypervisor may execute within a host operating system220executing on the virtual server202. Virtual machines240may then execute at a level above the hypervisor230. The Type 2 hypervisor may execute within the context of a host operating system220such that the Type 2 hypervisor interacts with the host operating system220. One or more virtual server202in a spatial simulation system200may instead include a Type 1 hypervisor (not shown.) A Type 1 hypervisor may execute on a virtual server202by directly accessing the hardware and resources within the hardware layer210. That is, while a Type 2 hypervisor230may access system resources through a host operating system220, as shown, a Type 1 hypervisor may directly access all system resources without the host operating system220. A Type 1 hypervisor230may execute directly on one or more physical processors316of virtual server202, and may include program data stored in the physical memory318.

The spatial simulation runtime248may cause the hypervisor230to create one or more virtual machines240in which additional spatial simulation runtime248and worker249instances may execute within guest operating systems246. Hypervisor230may load a virtual machine image to create a virtual machine240. The hypervisor230may execute a guest operating system246within virtual machine240. Virtual machine240may execute guest operating system246.

In addition to creating virtual machines240, hypervisor230may control the execution of at least one virtual machine240. Hypervisor230may present at least one virtual machine240with an abstraction of at least one hardware resource provided by the virtual server202(e.g., any hardware resource available within the hardware layer210.) Hypervisor230may control the manner in which virtual machines240may access physical processors216available in virtual server202. Controlling access to physical processors216may include determining whether a virtual machine240should have access to a processor216, and how physical processor capabilities are presented to the virtual machine240.

As shown inFIG. 2, virtual server202may host or execute one or more virtual machines240. A virtual machine240is a set of executable instructions that, when executed by a processor216, imitate the operation of a physical computer such that the virtual machine240may execute programs and processes much like a physical computing device. WhileFIG. 2illustrates an embodiment where a virtual server202hosts two virtual machines240, in other embodiments virtual server202may host any number of virtual machines240. Hypervisor230may provide each virtual machine240with a unique virtual view of the physical hardware, memory, processor, and other system resources available to that virtual machine240. Optionally, hypervisor230may provide each virtual machine240with a substantially similar virtual view of the physical hardware, memory, processor, and other system resources available to the virtual machines240.

Each virtual machine240may include a virtual disk242a-242n(generally242) and a virtual processor244a-244n(generally244.) The virtual disk242may be a virtualized view of one or more physical disks212of the virtual server202, or may be a portion of one or more physical disks212of the virtual server202. The virtualized view of the physical disks212may be generated, provided, and managed by the hypervisor230. Hypervisor230may provide each virtual machine240with a unique view of the physical disks212. Thus, the particular virtual disk242included in each virtual machine240may be unique when compared with the other virtual disks240.

A virtual machine240a-240nmay execute, using a virtual processor244a-244n, one or more workers249a-249nusing a guest operating system246a-246n. The guest operating system246may be any one of the following non-exhaustive list of operating systems: WINDOWS, UNIX, LINUX, iOS, ANDROID, SYMBIAN. Guest operating system246may be a purpose-built operating system based on one or more of the aforementioned operating systems. For example, guest operating system246may consist of a purpose-built version of LINUX which may comprise only the functional modules necessary to support operation of the workers249. Optionally, and as described in further detail below, a virtual machine240a-240nmay execute one or more bridge modules (not shown) corresponding to the one or more workers249a-249nexecuting in the virtual machine240a-240n. A virtual machine240a-240nmay also host one or more chunk modules (not shown), a receptionist module (not shown), and an oracle module (not shown.)

FIG. 2illustrates just one example of a spatially-optimized simulation system that may be used, and those of skill in the art will appreciate that the specific system architecture and computing devices used may vary, and are secondary to the functionality that they provide, as further described herein.

Referring toFIG. 3, some aspects described herein may be implemented in a cloud-based environment.FIG. 3illustrates an example of a spatially-optimized simulation environment (e.g., a development environment) based on a cloud-based computing platform system300. As shown inFIG. 3, client computing devices340a-340n(generally340) may communicate via the Internet330to access the spatially-optimized simulation executing on the virtual servers202(e.g., spatial simulation runtime248, server workers249, bridge modules (not shown), chunk modules (not shown), receptionist module (not shown), and an oracle module (not shown)) of the cloud-based computing platform310.

The spatial simulation runtime248contains the program code to implement the elements and components which comprise the spatially-optimized simulation environment, as described in further detail herein. For example, the spatial simulation runtime248may comprise implementation code for one or more of the bridge modules, chunk modules, receptionist module, and oracle module of the cloud-based computing platform310, as further described herein and as illustratively shown inFIG. 11, as well as provide worker management functions (starting processes, stopping processes, etc.). Additionally and alternatively, the spatial simulation runtime248may also expose an application programming interface (API) which may be utilized to monitor status, instantaneously and/or periodically, of the spatially-optimized simulation environment. The monitoring API may also be utilized to debug the status and behavior of the spatially-optimized simulation environment. In an illustrative embodiment, the spatial simulation runtime248may be implemented as a JAR (Java ARchive).

The cloud-based computing platform310may comprise private and/or public hardware and software resources and components. For example, a cloud may be configured as a private cloud to be used by one or more particular customers or client computing devices340and/or over a private network. Public clouds or hybrid public-private clouds may be used by other customers over open or hybrid networks. Known cloud systems may alternatively be used, e.g., MICROSOFT AZURE (Microsoft Corporation of Redmond, Wash.), AMAZON EC2 (Amazon.com Inc. of Seattle, Wash.), GOOGLE COMPUTE ENGINE (Google Inc. of Mountain View, Calif.), or others.

The spatially-optimized simulation development environment300may be deployed as a Platform-as-a-Service (PaaS) cloud-based computing service which may provide a platform for allowing a user to develop, run, and manage a spatially-optimized simulation. This may allow a user or client to create a spatially-optimized simulation without understanding the intricacies of distributed computation or requiring access to infrastructure teams or supercomputers. The spatially-optimized simulation development environment300may be delivered as a public cloud service from a provider. In such a scenario, client organizations may provide pre-existing models, simulations, and/or databases which may be integrated with the spatially-optimized simulation development environment300. Alternatively, the spatially-optimized simulation development environment may be delivered as a private service within a private network of a client organization.

The cloud-based computing platform310may comprise one or more virtual servers202a-202f(generally202) such as the virtual server202illustrated inFIG. 2. Optionally, the cloud-based computing platform310may comprise special-purpose virtual and/or physical computing resources which may be configured to provide spatially-optimized simulation functionality as described herein. AlthoughFIG. 3illustrates six virtual servers202(i.e.,202a-2020, those of skill in the art will appreciate that cloud-based computing platform310may comprise any number of virtual servers202. The virtual servers202may be interconnected via one or more networks in a manner that may allow each virtual server202to communicate directly with any other virtual server202in the cloud-based computing platform310in a peer-to-peer fashion. Optionally, virtual servers202may be arranged into a plurality of clusters of virtual servers. For example, clusters of virtual servers may be arranged based on a physical location of the physical computing resources used by the cloud-based computing platform310. In such an example, one cluster may be a first cloud datacenter located in California, and another cluster may be a second cloud datacenter located in Ireland (these are merely illustrative locations). In another example, clusters of virtual servers may be arranged based on an allocation to a spatially-optimized simulation. In such a scenario, one cluster may be comprised by a first subset of virtual servers202allocated to a first spatially-optimized simulation and another cluster may be a second subset of virtual servers202allocated to a second spatially-optimized simulation. A virtual server202may be manually or dynamically reassigned to a different cluster if or when the virtual server202is moved or if or when the computing resource requirements for the first spatially-optimized simulation and the second spatially-optimized simulation may change over time. Client computing devices340connecting to a virtual server202may be unaware of which cluster, if any, the virtual server202belongs to and may also be unaware whether the virtual server202may change membership from one cluster to another during the course of the connection.

The cloud-based computing platform system300may also comprise a cloud-based data store320. The storage resources in the cloud-based data store320may include storage disks (e.g., solid state drives (SSDs), magnetic hard disks, etc.) and other storage devices. Alternatively, the cloud-based data store320may be provided by a known cloud-based storage provider, such as, AMAZON S3 (Amazon.com Inc. of Seattle, Wash.), GOOGLE CLOUD STORAGE (Google Inc. of Mountain View, Calif.), or others. Optionally, the cloud-based data store320may be implemented or deployed separately from cloud-based computing platform310as shown inFIG. 3. Optionally, the cloud-based data store320may be implemented or deployed within the cloud-based computing platform310. For example, both the cloud-based computing platform310and the cloud-based data store320may be provided by a cloud systems provider as part of the resources assigned to the cloud system by the provider.

The cloud-based data store320may comprise one or more application assemblies322. An application assembly322may comprise data which may define entities and components of a spatially-optimized simulation, as well as, procedures which may define one or more behaviors of each of the entities and components in a spatially-optimized simulation. Optionally, an application assembly322may comprise schemas, data structures, serialized objects, and the like which may define the entities and components which make up a spatially-optimized simulation. Optionally, an application assembly322may comprise computer-readable code or instructions, scripts, statically-linked libraries, dynamically-linked libraries, and the like which may define one or more behaviors for the elements in the spatially-optimized simulation. Virtual servers202in the cloud-based computing platform310may load an application assembly from the cloud-based data store320. The spatial simulation runtime248in each virtual server202may use the data and procedures comprised in an application assembly322to cause the execution of a distributed, persistent, and spatially-optimized simulation. The cloud-based data store320may also comprise initialization data and/or procedures324which define a starting or initial condition for a spatially-optimized simulation. For example, the cloud-based computing platform310may load initialization data324from the cloud-based data store320which may cause a predetermined number of entities and components to be instantiated and initialized to a predetermined initial state. In another example, the cloud-based computing platform310may load and may execute one or more initialization procedures324which may cause a predetermined number of entities and components to be instantiated and initialized to a predetermined state. In yet another example, the entities and the components may be instantiated and initialized to a predetermined state based on a combination of initialization data324and initialization procedures324loaded by the cloud-based computing platform310from the cloud-based data store320.

The cloud-based data store320may comprise a snapshot326of a simulation. A simulation snapshot326may define a valid state of a simulation, and may comprise data and/or procedures which may return a spatially-optimized simulation to that valid state if or when it is loaded and/or executed by the cloud-based computing platform310from the cloud-based data store320. The valid simulation state defined by snapshot326may be a known state or a desired state of the simulation. Optionally, the simulation state defined by snapshot326may be a previously saved state of a running simulation.

A portion of the cloud-based computing platform310may be related, for example, one or more virtual servers202may be executing a spatially-optimized simulation on behalf of the same end user, or on behalf of different users affiliated with the same company or organization. In other examples, certain virtual servers202may be unrelated, such as users affiliated with different companies or organizations. For unrelated clients, information on the virtual servers202or cloud-based data store320of any one user may be hidden from other users.

In some instances, client computing devices340may implement, incorporate, and/or otherwise include one or more aspects of computing device101and computing device202. Client computing devices340may be any type of computing device capable of receiving and processing input via one or more user interfaces, providing output via one or more user interfaces and communicating input, output, and/or other information to and/or from one or more other computing devices. For example, client computing devices340may be desktop computers, laptop computers, tablet computers, smart phones, or the like. In addition, and as illustrated in greater detail below, any and/or all of client computing devices340may, in some instances, be special-purpose computing devices configured to perform specific functions.

The client computing devices340may comprise a worker integration library342and an instance of a worker process249. A client computing device340may utilize the worker integration library342and the worker process249to connect to a spatially-optimized simulation executing in the cloud-based computing platform310. As described in further detail below, a client computing device340may receive data from the cloud-based computing platform310describing relevant portions of the spatially-optimized simulation. The worker process249executing in the client computing device340may utilize that received data to render the relevant portions of the spatially-optimized simulation on a display or other user interface device. The client computing device340may also transmit data and commands to cloud-based computing platform310which may affect the state of the spatially-optimized simulation. The data and commands may be transmitted in response to user input. Optionally, the transmitted data and commands may be generated in response to calculations performed by the worker integration library342or the worker process249.

Advantageously, and as illustrated in greater detail above, a simulation developer using a spatially-optimized simulation development environment may be able to scale up a game or simulation to be considerably larger than would be possible using a single machine. In addition, the spatially-optimized simulation development environment may allow for an arbitrary number of user participants and data sources to integrate into the simulation. Furthermore, the spatially-optimized simulation development environment may remove the need for a simulation developer to worry about scalability or data synchronization among different parts of the spatially-optimized simulation.

FIG. 3illustrates just one example of a spatially-optimized simulation development environment that may be used, and those of skill in the art will appreciate that the specific system architecture and computing devices used may vary, and are secondary to the functionality that they provide, as further described herein.

FIG. 4illustrates one example of a block diagram of a spatially-optimized simulation that may be implemented according to one or more illustrative examples of the disclosure. A spatially-optimized simulated world410may comprise a collection of entities (e.g., entity1420, entity2430, and entity N430.) An entity may represent a fundamental computational unit or other unit of simulated world410. WhileFIG. 4illustrates a simulated world410comprising three entity types, in other examples, a simulated world410may comprise any number of entity types. Additionally, simulated world410may comprise any number of instances of each entity type. For example, in a city simulation, simulated world410may comprise a car entity, a pedestrian entity, a traffic signal entity, a road entity, a building entity, and the like. In such a scenario, the city simulation may comprise large and different quantities of instances of each entity. In another example, in a video game world simulation, simulated world410may comprise a monster entity, a player entity, a weapon entity, a tree entity, a rock entity, and the like. The video game simulated world may comprise a handful of instances of the monster entity, one player entity instance for each player active in the game, and potentially millions of instances of the tree and rock entities. In yet another example, in a trading simulation, simulated world410may comprise a trader entity, a stock entity, a mutual fund entity, a market agent entity, and the like. The simulated trading world may comprise small numbers of trader and market agent entities and may also comprise thousands of stock and mutual fund entities.

The state and behavior of an entity (e.g.,420,430, and440) may be determined by the combination of components (e.g.,421,422,423,431,432,433, and441) comprised by the entity. Each component (e.g.,421,422,423,431,432,433, and441) may comprise a subset of the state and behavior attributed to the entity (e.g.,420,430, and440) as a whole. For example, as shown inFIG. 4, entity1420may comprise component A421, component B422, and component C423; entity2430may comprise component A431, component D432, and component E433; and entity N440may comprise component F441. As will be appreciated by one of skill in the art, the number and types of components comprised by any one entity may be arbitrary and not limited to the example illustrated inFIG. 4. Optionally, two or more entities may comprise different instances of a particular component if or when the two or more entities have a set of properties and behaviors in common. For example, entity1420may represent a rock in a video game simulation and entity2430may represent a monster in the same simulation. Both entities (i.e.,420and430) may share a component A (e.g.,421and431) which may define the properties and behaviors for a rigid body, i.e., mass and velocity.

Entities (e.g.,420,430, and440) may comprise properties which may be common across all entities. For example, entities (e.g.,420,430, and440) may comprise an identifier value which may be used to uniquely identify each entity instance within simulated world410. Entities (e.g.,420,430, and440) may comprise properties which may be shared across multiple components. For example, entities (e.g.,420,430, and440) in a video game simulation may comprise position and velocity values since it is likely that most components in such a simulation may require access to those values. Additionally, locating commonly used properties within an entity may reduce coupling between the components and facilitate communication between the components of an entity.

Referring toFIG. 5, some aspects described herein may be implemented, incorporated, and/or otherwise included by one or more components421,422,423,431,432,433, and441.FIG. 5illustrates an example implementation of a component510in a spatially-optimized simulation system as described herein. A component510may comprise a collection of related persistent properties530a-530n(generally530) and events550a-550z(generally550.) The component510may also comprise procedures540which may change the value of the component's properties and may generate events. Procedures540may execute, as part of a server worker249a-249n, in a server such as one of the servers illustrated inFIGS. 2-3(e.g.,240a-240n,202a-202f, and340a-340n.) A spatial simulation runtime248or other software entity may delegate the write authority of the properties and event generation from the component510to a specialized worker560. Other components and/or workers executing within a spatially-optimized simulation may cause or trigger updates in the state of component510via commands520a-520m(generally520.) Alternatively, no delegation may take place.

Components may comprise one or more properties530. The state of a component510may be defined by the values held by the properties530comprised by the component510. Similarly, the state of an entity may be defined by the values held by the properties530of all the components comprised by the entity. The state of a component510may be stored in local memory (e.g.,242a-242n,244a-244n,218) for access during execution of the spatially-optimized simulation. Optionally, the state of a component510may be stored in cloud-based data store320as part of a snapshot326and thus may be persisted across simulation runs. The state of a component510may be stored periodically (e.g., continuously.) The rate at which the state of a component510is persisted may vary based on one or more factors. For example, if or when the state of a component510changes rapidly, the storage rate may also increase commensurate with the rate of change. In another example, the storage rate may be higher for properties which may require a higher degree of accuracy than other properties.

Where it is described that an entity or component may exhibit a certain behavior, it is to be understood that another element, such as a worker module, for example, may perform the required calculations on behalf of that entity or component and emit or receive the corresponding signals or data.

Events550may indicate the occurrence of a transient action on component510. Component510may emit one or more events550in response to making a determination (or events550may be emitted for one or more components510), reaching a particular result, receiving user input, or another type of trigger. Other components within the spatially-optimized simulation may monitor the occurrence of an event550and update their state or perform an action in response to the event550. The other components may be comprised by the same entity (e.g., a worker module) as the emitting component or may be comprised by other entities within the spatially-optimized simulation. For example, a traffic signal entity in a city simulation may emit an event if or when the traffic signal indicator changes to red. A vehicle entity in the city emulation may receive the event and may come to a stop in response to the event. In another example, a rigid body component may emit an event if or when it has determined that it has collided with another object.

Optionally, component510may comprise procedures540which may update the values of properties530, as well as, cause the component510to emit events550. Procedures540may also receive and process commands520from other components and/or the spatial simulation runtime248. Thus, procedures540may define the behavior of component510within the spatially-optimized simulation. Alternatively, a spatial simulation runtime248may delegate to a specialized worker560the implementation of the behavior of component510. In such a scenario, spatial simulation runtime248may delegate write access of properties530and events550from component510to specialized worker560. Component510may have at most one writer assigned to it at any one time. Thus, a spatial simulation runtime248may remove the ability of procedures540to modify properties530and emit events550until delegation to specialized worker560is revoked. Optionally, a specialized worker560may implement the behavior of a component based on real-time and/or real-world behavior of a physical entity being simulated. For example, a specialized worker560may periodically collect position, velocity, and direction data from one or more sensors mounted on a vehicle or other moving object and use that information to modify properties530and emit events550of component510. In another example, a specialized worker560may receive previously recorded real-world position, velocity, and direction data of a vehicle or other moving object and use that information to modify properties530and emit events550of component510. Thus, a specialized worker560may be used to incorporate real-time and/or real-world into the spatial simulation. Any other real world objects, people, events, and/or systems may be used to generate data as input for a simulation.

Delegation may require specification of a worker constraint which may identify a type of worker capable of simulating the behavior of component510. Worker560may be one of a plurality of worker types which may be specialized to perform certain kinds of computations. Specialized workers560may only understand a subset of the components (e.g.,421,422,423,431,432,433, and441) that define entities (e.g.,420,430, and440) within a spatially-optimized simulation410. For example, in a city simulation, one worker type may simulate vehicle positions, another worker type may simulate traffic signals, and yet another type may simulate environmental emissions.

Worker560may comprise data structures and/or objects and software programs to simulate the behavior of a subset of the components (e.g.,421,422,423,431,432,433, and441) within a spatially-optimized simulation410. Worker560may be a process corresponding to one or more aspects of workers249, as described inFIGS. 2 & 3. Thus, worker560may execute, as part of a server worker249a-249n, in a server such as one of the servers illustrated inFIGS. 2-3(e.g.,240a-240n,202a-202f, and340a-340n.) Worker560may read the properties530of any component (e.g.,421,422,423,431,432,433, and441) in spatially-optimized simulation410. However, worker560may only write the properties530of those components (e.g.,421,422,423,431,432,433, and441) that have delegated their write authority to worker560. A worker560may be said to be authoritative for a component510if or when component510has delegated its write authority to worker560. Worker560may be authoritative to a subset of entities (e.g.,420,430, and440) within a spatially-optimized simulation410. Optionally, worker560may be authoritative to one or more entities which may be located close to each other within spatially-optimized simulation410.

In order to simulate the behavior of a component (e.g.,421,422,423,431,432,433, and441), worker560may need information (e.g., properties, events) from nearby entities (e.g.,420,430, and440) within spatially-optimized simulation410. For example, a worker simulating a traffic intersection in a city simulation may need information from vehicles in nearby intersections, but not from vehicles which are miles away from the intersection. The interest region for worker560may comprise all regions comprising nearby entities (e.g.,420,430, and440) from which the worker560needs information. The interest region for worker560may comprise entities (e.g.,420,430, and440) for which worker560is not authoritative. The spatially-optimized simulation410may automatically synchronize the data between worker560and the other workers which are authoritative for the nearby entities.

Worker560may communicate with the spatially-optimized simulation410(e.g. with entities) via a bridge610, as illustrated inFIG. 6.FIG. 6illustrates an example implementation of a worker560communicating with a bridge610in a spatially-optimized simulation410as described herein. A bridge610may be responsible for communicating relevant information (e.g., properties, events) from worker560to other interested workers within a spatially-optimized simulation410. Bridge610may also be responsible for communicating relevant information from nearby entities within the interest region for worker560. Bridge610may be assigned to only one worker560and worker560may communicate with only one bridge610. That is, there may be a one-to-one relationship between bridge610and worker560. Bridge610may execute, as part of a server worker249a-249n, in a server such as one of the servers illustrated inFIGS. 2-3(e.g.,240a-240n,202a-202f, and340a-340n.)

Communication between bridge610and worker560may be effectuated via a worker application programming interface (API). Optionally, worker560may be wrapped by worker API wrapper630. Worker API wrapper may allow a worker560which may have been developed independently from the spatially-optimized simulation development environment to possibly function within and by managed by bridge610. Optionally, the worker API may allow for the integration of pre-existing non-distributed simulation programs into a large-scale distributed spatially-optimized simulation. For example, a game engine (e.g., UNITY by Unity Technologies SF of San Francisco, Calif.) may be integrated into a spatially-optimized simulation to simulate rigid-body physics or to provide client-side rendering and navigation. In another example, a multi-modal traffic flow simulation software package (e.g., open source MATSIM, or other commercially available software packages) may be integrated into a city spatially-optimized simulation. Other worker engines or programs may alternatively or also be used.

In another example implementation, specialized worker560may require special-purpose hardware or other physical resources that might not be available within a cloud-based platform310. In such a scenario, the worker API wrapper640and bridge610may reside on a computing device physically located remotely from the cloud-based platform310and may connect to the cloud-based platform310via the Internet or another type of network. Such a specialized worker560, which may reside outside of the cloud-based platform310, (e.g., may execute on client devices340a-340n) may be referred to as an external worker. And another specialized worker560, which may execute within the cloud-based platform310, (e.g., may execute on servers240a-240n,202a-202f) may be referred to as an internal worker. Any one or more of the features described with reference to the cloud-based platform310may be used in or with this example implementation.

The worker API may allow a bridge to add or remove entities from the interest region of a worker, notify a worker of component state changes, delegate a component to a worker or to remove the delegation, signal component state changes for components on which the worker is authoritative, among other related functionality as described herein.

Among the functions provided by the worker API may be functions for adding or removing an entity. Optionally, worker API wrapper630may comprise a handler method to be called by bridge610when an entity enters the interest region of worker560. For example, Method 1 is one example of a method signature that may be used to add an entity to the interest region of worker560.

eid is a value which may uniquely identify the entity being added; and

initialState is a data structure and/or object which may describe the initial state of the entity being added.

Although Method 1 is provided as an example for adding an entity to the interest region of worker560, various other methods and/or functions may be used. For instance, other parameters may be included in the method without departing from the disclosure. Method 1 may then be passed to a RegisterEntityAddHandler( ) worker API function, which may cause the Method 1 handler to be called whenever an entity should be added.

Optionally, worker API wrapper630may comprise a handler method to be called by bridge610when an entity leaves the interest region of worker560. For example, Method 2 is one example of a method signature that may be used to remove an entity from the interest region of worker560.

eid is a value which may uniquely identify the entity being removed.

Although Method 2 is provided as an example for removing an entity from the interest region of worker560, various other methods and/or functions may be used. For instance, other parameters may be included in the method without departing from the disclosure. Method 2 may then be passed to a RegisterEntityRemoveHandler( ) worker API function, which may cause the Method 2 handler to be called whenever an entity should be removed.

The worker API may also comprise functions for notifying a worker that the properties of a component within the worker's interest region have changed state. For example, worker API wrapper630may comprise a handler method to be called by bridge610when the properties of a component within the interest region of worker560have changed state. Method 3 is one example of a method signature that may be used to notify worker560of the changed state.

eid is a value which may uniquely identify the entity which may comprise the component whose properties changed state; and

state is a data structure and/or object which may describe the state of the component.

Although Method 3 is provided as an example for notifying worker560of a changed state, various other methods and/or functions may be used. For instance, other parameters may be included in the method without departing from the disclosure. In some variants, the state parameter may comprise only the subset of properties of the component that have changed since the last update, for efficiency. Method 3 may then be passed to a AddComponentStateChangeHandler( ) worker API function, which may cause the Method 3 handler to be called whenever the properties of a component within the worker's interest region have changed state.

Among the functions provided by the worker API may be functions for dynamically changing component authority assignments. Worker API wrapper630may comprise a handler method to be called by bridge610when worker560may now be authoritative for a component. For example, Method 4 is one example of a method signature that may be used to delegate component authority to worker560.

eid is a value which may uniquely identify the entity which may comprise the component being delegated; and

cid is a value which may uniquely identify the component being delegated.

Although Method 4 is provided as an example for delegating component authority to worker560, various other methods and/or functions may be used. For instance, other parameters may be included in the method without departing from the disclosure. Method 4 may then be passed to a RegisterComponentDelegateHandler( ) worker API function, which may cause the Method 4 handler to be called whenever worker560may now be authoritative for a component.

Optionally, worker API wrapper630may comprise a handler method to be called by bridge610when worker560may no longer be authoritative for a component. For example, Method 5 is one example of a method signature that may be used to remove delegation authority for a component from worker560.

eid is a value which may uniquely identify the entity which may comprise the component being undelegated; and

cid is a value which may uniquely identify the component being undelegated.

Although Method 5 is provided as an example for removing delegation authority for a component from worker560, various other methods and/or functions may be used. For instance, other parameters may be included in the method without departing from the disclosure. Method 5 may then be passed to a RegisterComponentUndelegateHandler( ) worker API function, which may cause the Method 5 handler to be called whenever worker560may no longer be authoritative for a component.

In yet other examples, worker API wrapper630may comprise a handler method to be called by bridge610for setting or unsetting a worker560as authoritative for a component. For example, Method 7 is one example of a method signature that may be used to set or remove delegation authority for a component for worker560.

eid is a value which may uniquely identify the entity which may comprise the component;

cid is a value which may uniquely identify the component; and

isAuthoritative is a true/false value which may indicate whether to set or unset worker560as authoritative for a component.

Although Method 6 is provided as an example for setting or unsetting a worker560as authoritative for a component, various other methods and/or functions may be used. For instance, other parameters may be included in the method without departing from the disclosure.

The worker API may also comprise functions for notifying other workers that the properties of a component for which worker560is authoritative have changed state. For example, worker API wrapper630may comprise a method to be called by worker API wrapper630when the properties of a component for which worker560is authoritative have changed state. Method 7 is one example of a method signature that may be used to update the properties of the components for which worker560is authoritative.

eid is a value which may uniquely identify the entity which may comprise the component whose properties changed state; and

state is a data structure and/or object which may describe the updated state of the component.

Although Method 7 is provided as an example for updating the properties of the components for which worker560is authoritative, various other methods and/or functions may be used. For instance, other parameters may be included in the method without departing from the disclosure. Method 7 may be called whenever the properties of a component for which worker560is authoritative have changed state.

Optionally, worker560may be configured to periodically send a heartbeat signal to bridge610. If or when worker560ceases to transmit heartbeat signals, bridge610may determine that worker process560may have terminated unexpectedly. In response to the determination, bridge610may terminate cleanly and request that a replacement worker process560(and new counterpart bridge610) be allocated and instantiated.

FIG. 7depicts a flowchart that illustrates a method of registering a worker process with a spatially-optimized simulation. The algorithm shown inFIG. 7and other similar examples described herein may be performed in a computing environment such as the system illustrated inFIGS. 3-6, as well as other systems having different architectures (e.g., all or part ofFIGS. 1-2.) The method illustrated inFIG. 7and/or one or more steps thereof may be embodied in a computer-readable medium, such as a non-transitory computer readable memory.

Referring toFIG. 7, step702, a spatial simulation runtime248may have instantiated and provisioned a server worker process249on a virtual server202based on a determination that a new worker560instance was needed. For example, spatial simulation runtime248may have detected increased simulation workload which may require an additional worker560instance to be created. In another example, spatial simulation runtime248may have detected that a pre-existing worker instance may have crashed and must be replaced by a new worker560instance. In step704, worker560may send a message or otherwise signal to bridge610that worker560is ready to accept simulation work. In response to the receipt of the ready message from worker560, bridge610may announce to the spatially-optimized simulation410that worker560is ready to accept simulation work, as shown in step706. A spatial simulation runtime248may, in step708, cause bridge610to add one or more entities to the interest region of worker560. For example, bridge610may call an OnEntityAdd method in worker API wrapper630to add each of the one or more entities. A spatial simulation runtime248may, in step710, cause bridge610(or other entity or module) to delegate write authority of one or more components to worker560. For example, bridge610may call an OnComponentDelegate method in worker API wrapper630to delegate authority for each of the one or more components. In step712, bridge610may notify worker560that one of the components within the interest region for worker560has changed state. For example, bridge610may call an OnStateChanged_Component1 method in worker API wrapper630to notify worker560of the change in the state of the component. Worker560may recalculate the state for each of the one or more components for which is it authoritative. If or when worker560determines, in step714, that the state of a component has changed, then worker API wrapper630may call an UpdateState_Component1 method for each one of the one or more components whose state has changed, as shown in step716, and the method ends. If or when worker560determines, in step714, that none of the components have changed state, then worker560may return to step712and wait for another notification of a change in the state of the component in the interest region for worker560.

FIG. 8Aillustrates one example of a spatially-optimized simulated world800that may be implemented according to one or more illustrative embodiments of the disclosure. As shown inFIG. 8A, spatially-optimized simulated world800may be sub-divided into a plurality of chunks or regions. AlthoughFIG. 8Aillustrates a spatially-optimized simulated world800and chunks using two dimensions, those of skill in the art will appreciate that a spatially-optimized simulated world800and chunks may comprise one or more dimensions, as may be specified by a simulation developer. For example, a one-dimensional spatially-optimized simulated world800may be represented by a line and chunks may comprise portions of the line. In another example, spatially-optimized simulated world800may simulate a three-dimensional (3D) world and chunks may comprise three-dimensional portions (e.g., sphere, cube, etc.) of the 3D simulated world. Each chunk may be controlled by a chunk actor (e.g., chunk module), which is the program, process, routine, or agent responsible for content within that chunk.

Each chunk actor810aa-810nn(generally810) may be allocated to a chunk server (e.g.,820a-820c) such as the server illustrated inFIGS. 2-3(e.g.,240a-240n,202a-202f), as well as other systems having different architectures (e.g. all or part ofFIG. 1.) A subset of chunk actors810may be allocated to the same chunk server such that the functionality associated with the subset of chunk actors810may be provided by the chunk server. For example, chunk actors810aa,810ab,810ba, and810bbmay be allocated to a chunk server820a; chunk actors810ac,810ad,810ae,810bc,810bd,810be,810cc,810cd, and810cemay be allocated to a chunk server820b; and chunk actor810afmay be allocated to a chunk server830c. Chunk server allocation of chunk actors810may be based on a number of factors. Chunk actors810may be allocated to a chunk server820based on their relative locations. For example, chunk actors810aa,810ab,810ba, and810bbmay be allocated to a chunk server820abased on their being adjacent to each other. Such an allocation may be advantageous as it may reduce network traffic and latency as adjacent chunk actors are more likely to communicate with each other than chunk actors810which are located far away for each other. Optionally, chunk actors810may be allocated to a chunk server820based on chunk processing workload and available server processing resources. For example, the processing load of a chunk actor810may be determined based on the number of entities comprised within its chunk or region. Based on such a determination, chunk actors810may be allocated to a chunk server820until a predetermined indication of server load is achieved. In such a scenario, chunk actors810(i.e.,810ac,810ad,810ae,810bc,810bd,810be,810cc,810cd, and810ce) may be allocated to a chunk server820buntil chunk server820breaches a predetermined server load value. In yet other examples, each chunk server820may be allocated a predetermined number of chunk actors. For example, chunk server820cmay be allocated one chunk actor810(i.e.,810af.)

As a spatially-optimized simulation800progresses, the location and quantity of entities represented within the simulated world may change. As shown inFIG. 8B, the chunks or regions assigned to chunk actors850a-850nmay change in size, shape, and quantity as needed based on the instantaneous state of the spatially-optimized simulation800. Optionally, chunk actor850may be assigned to a portion of a spatially-optimized simulated world800based on the location, quantity, and density of entities within the assigned chunk. For example, the size of a chunk assigned to chunk actor850may be reduced as the number of entities migrating to the assigned chunk increases. For example, the chunks assigned to chunk actors850fand850gmay comprise a higher density of entities than the chunks assigned to chunk actor850d. In other examples, the size and shape of a chunk region may be determined based on the workload associated with the entities located within the chunk region. For example, the processing load, or another value indicative of the workload, of a chunk actor850may be determined based on the number and/or type of entities and components comprised within its chunk or region. Based on such a determination, the size and shape of a chunk or region allocated to chunk actors850may be adjusted until a predetermined indication of workload is achieved. In yet other examples, the size, shape, and quantity of the chunks assigned to chunk actors850may remain unchanged as the spatially-optimized simulation800progresses. For example, chunk actors and their assigned regions may remain as shown inFIG. 8A. Server allocation of chunk actors850may also vary (not shown) based on similar factors as those discussed in detail above with reference toFIG. 8A.

Additionally, chunk actors may be logically grouped into chunk actor layers.FIG. 8Cillustrates one example of a spatially-optimized simulated world comprising chunk actor layers870,880, and890. The spatially-optimized simulated world illustrated inFIG. 8Cmay be similar to the spatially-optimized simulated world illustrated inFIGS. 8A-8B. Each chunk actor layer (e.g.,870,880, and890) may comprise one or more chunk actors. For example, chunk actor layer870may comprise chunk actors871a-871m, chunk actor layer880may comprise chunk actors881a-881m, and chunk actor layer890may comprise chunk actors891aa-891gg. Those of skill in the art will appreciate that chunk actors871,881, and891may be grouped into any number of chunk actor layers and that each chunk actor layer may comprise different amounts of chunk actors and that the number of chunk actors comprised in each layer may vary as spatially-optimized simulation progresses, as described in detail above. Chunk actor layers870,880, and890may overlap each other and share at least a portion of the spatially-optimized simulated world. For example, as shown inFIG. 8C, chunk actor layers870,880,890cover and share all regions defined by the spatially-optimized simulated world800. In this example, there may be three chunk actors (or two or one) for the same region of the virtual world.

Chunk actors871,881, and891may be organized into one or more chunk actor layers (e.g.,870,880, and890) based on one more criteria. Optionally, a chunk actor layer (e.g.,870,880, and890) may comprise chunk actors (e.g.,810,850,871,881, and891) which may be configured to store the canonical data of one particular type of component in the spatially-optimized simulated world. For example, chunk actor layer870may comprise chunk actors871a-871mwhich may store the properties and state information for component A421and431as illustrated inFIG. 4. Similarly, chunk actor layer880may comprise chunk actors881a-881mwhich may store the properties and state information for component B422as illustrated inFIG. 4. Spatially-optimized simulated world800may comprise one chunk actor layer (e.g.,870,880, and890) for every component type (e.g.,421-423,432-433, and441) comprised in the spatially-optimized simulated world800.

Optionally, a chunk actor layer (e.g.,870,880, and890) may comprise chunk actors (e.g.,810,850,871,881, and891) which may comprise entities of similar size. For example, chunk actor layer890may comprise chunk actors891which may manage entities which may be small in size. Chunk actor layer880may comprise chunk actors881which may manage entities which may be generally larger than the entities in chunk actor layer890. Additionally, chunk actor layer880may comprise a coarse-grained representation of the entities comprised by chunk actor layer890. Chunk actor layer870may comprise chunk actors871which may manage entities which may be generally larger than the entities in chunk actor layer880. Additionally, chunk actor layer870may comprise a coarse-grained representation of the entities comprised by chunk actor layer880.

In yet other examples, chunk actors (e.g.,810,850,871,881, and891) may be grouped into chunk actor layers (e.g.,870,880, and890) based on the importance of the entities comprised by the chunk actors. For example, entities with higher importance may be grouped into higher level layers. Optionally, the spatially-optimized simulated world800may comprise a single chunk actor layer which may comprise all chunk actors. In such a scenario, each chunk actor may be responsible for all entities located within the region monitored by the chunk actor.

A chunk actor (810,850,871,881, and891) may monitor a set of entities which are assigned to the chunk actor and determine that an entity may need to be transferred to another chunk. For example, an entity may need to be migrated to a second chunk actor if or when the entity has moved to a region assigned to the second chunk actor. The chunk actor (810,850,871,881, and891) may determine the second chunk actor based on the current position of the entity. For example, a chunk actor (810,850,871,881, and891) may use an algorithm or mathematical expression to map the entity's position to a chunk actor region. Alternatively or additionally, the chunk actor may obtain a mapping of the entity's position. For example, snapshot326may comprise a map of the chunk actors in the spatially-optimized simulated world800and chunk actor (810,850,871,881, and891) may obtain the map to determine the identity of the second chunk actor using the current position of the entity. In another example, the spatially-optimized simulated world800may comprise a distributed hashtable or distributed data structure which may maintain a mapping from a position in the spatially-optimized simulated world800to its corresponding chunk actor. The chunk actor (810,850,871,881, and891) may query the distributed data structure and may obtain an indication of the second chunk actor.

An entity (or software simulating or representing an entity) may monitor its position or other attributes within the spatially-optimized simulated world800and determine whether it needs to migrate from its current chunk to another chunk. For example, an entity may change position from within the chunk of a first chunk actor (810,850,871,881, and891) to the chunk assigned to a second chunk actor (810,850,871,881, and891.) The entity may determine the second chunk actor based on the current position of the entity. For example, the entity may use an algorithm or mathematical expression to map the entity's position to a chunk actor region. Alternatively or additionally, the entity may obtain a mapping of the entity's position. For example, the entity may obtain the mapping from snapshot326or, in another example, the entity may obtain the mapping from a distributed hashtable or distributed data structure. If or when an entity determines the need to migrate to a second chunk actor, then the entity may notify its current chunk actor and request to be migrated to the second chunk actor.

A first chunk actor may migrate an entity to a second chunk actor by communicating directly, in a peer-to-peer fashion, with the second chunk actor. The first chunk actor may forward the entity's state information to the second chunk actor and the second chuck actor may start monitoring and receiving state change notifications for the migrated entity. The first chunk actor may also stop monitoring and receiving state change notifications from the migrated entity.

A chunk actor (810,850,871,881, and891) may monitor and receive state change notifications from all assigned components for all the entities located within its corresponding chunk. The chunk actor (810,850,870,880, and890) may store the states of the assigned components in the local memory of the chunk server820allocated to the chunk actor for access during execution of the spatially-optimized simulation. In some examples, the states of the assigned components may be stored in cloud-based data store320as part of a snapshot326and thus may be persisted across simulation runs. The snapshot326may also be used to restore a chunk server320if or when the chunk server320has terminated unexpectedly.

The rate at which components emit state change notifications and the rate at which the state changes are stored may be determined by one of a multiple of data policies implemented by the chunk actor (810,850,871,881, and891.) State change notifications may be emitted based on the distance between the emitting and the receiving entities. If or when the receiving entity is a large distance away from the emitting entity, the emitting entity may publish state changes at a slower rate. Additionally, the emitting entity may reduce the period of time between state change notifications if or when the receiving entity is closer. In such a scenario, the emitting entity may calculate or determine the state at the same rate; the calculation rate may be unaffected by the distance changes. Thus, allowing an entity to publish state changes at varying rates to multiple receiving entities.

The publishing rate may be determined based on overlap of interest regions.FIG. 9illustrates one example of a representation of the interest region of a worker that may be implemented according to one or more illustrative embodiments of the disclosure. Interest regions A, B, C, and D (i.e.,910a-910d) may represent the interest regions for workers A, B, C, and D (not shown), respectively, within a spatially-optimized simulated world900. Workers A, B, C, and D may incorporate and/or otherwise include one or more aspects of worker560illustrated inFIGS. 5-7. AlthoughFIG. 9illustrates interest regions910using two dimensions, those of skill in the art will appreciate that interest regions may comprise up to as many dimensions as are simulated by the spatially-optimized simulated world. For example, interest regions910may comprise three dimensions in a 3D simulated world.

An interest region may overlap with one or more other interest regions. As shown inFIG. 9, interest region A910amay overlap with interest region B910bin regions930a, with interest region C910cin region930d, and with interest region D910din region930e. Similarly, interest region B910bmay overlap with interest region C910cin region930c. Interest regions A, B, and C (i.e.,910a-910c) may all overlap in region930b. Regions920a-920dmay indicate regions with no overlap. In other examples, not shown, various interest regions may overlap or not in various other combinations.

An entity (e.g.,940a-940c) may publish (or have published on its behalf) state change notifications at a low rate if or when the entity is located within a portion of its authoritative worker's interest region that does not overlap with any other worker's interest region. For example, entity940amay publish state change notifications at a slower rate if or when it may be located within region920a. An entity (e.g.,940a-940c) may publish state change notifications at a medium rate if or when the entity is located within a portion of its authoritative worker's interest region that does overlap with any other worker's interest region. For example, entity940cmay publish state change notifications at a normal rate if or when it may be located within region930e. An entity (e.g.,940a-940c) may publish state change notifications at a high rate if or when the entity is located within a portion of its authoritative worker's interest region that does overlap with two or more other worker's interest region. For example, entity940b(or a worker) may publish state change notifications at a higher rate if or when it may be located within region930b.

Alternatively, an entity may implement multiple separate components which publish their properties at different rates. For example, a vehicle entity may implement a high-fidelity position component that publishes the vehicle's position at a high rate, and a second low-fidelity position component that publishes the vehicle's position at a low rate. Other entities may choose to monitor either the high-fidelity or low-fidelity component.

FIG. 10illustrates another example of a representation of the interest region of a worker that may be implemented according to one or more illustrative embodiments of the disclosure. Interest regions A, B, C, and D (i.e.,910a-910d) may represent the interest regions for workers A, B, C, and D (not shown), respectively, within a spatially-optimized simulated world900. Workers A, B, C, and D may incorporate and/or otherwise include one or more aspects of worker560illustrated inFIGS. 5-7 and 9. AlthoughFIG. 10illustrates interest regions910using two dimensions, those of skill in the art will appreciate that interest regions may comprise up to as many dimensions as are simulated by the spatially-optimized simulated world. For example, interest regions910may comprise three dimensions in a 3D simulated world.

Optionally, a worker process (e.g.,560, workers A, B, C, and D) may periodically determine a load metric. The load metric may be a value indicative of the instantaneous workload on the worker process and its ability to perform additional simulation computation work. For example, a load metric may consist of a value between 0 and 1 where a value of 1 may indicate a worker which is unable to accept additional work. A worker process (e.g.,560, workers A, B, C, and D) may periodically transmit its load metric to chunk actor(s) responsible for the chunk region(s) covered by the worker's interest region. A worker process (e.g.,560, workers A, B, C, and D) may periodically calculate a load density center (e.g.,1050a-1050d). The load density centers1050may represent a center of mass for interest region910wherein the “mass” relates to the computation workload of the worker process (e.g.,560, workers A, B, C, and D.) For example, interest region910amay comprise a load density center1050abased on the location, quantity, and processing load of the entities and components assigned to worker A. Load density centers1050need not be in a geometric center of interest regions910. Worker processes (e.g.,560, workers A, B, C, and D) may update the location of their respective load density centers1050as spatially-optimized simulation900progresses and the location, quantity, density, and processing load requirements of the entities in the simulation900change. Load density may also be described as the processing requirement needed to simulate or represent a unit of space, area or other portion of the virtual simulation or world. A load density center may also be described as a mean position of computational requirements for a particular body, or portion of the virtual simulation or world, for example.

A chunk actor (e.g.,810,850,871,881, and891) may monitor the load metrics and load density centers reported by the workers within its chunk region. Based on the monitoring, a chunk actor may determine whether a worker may be at or over maximum processing capacity. Based on the determination, the chunk actor may attempt to reduce the worker's processing workload. In one example, the chunk actor may remove delegation authority for one or more entities from the worker process, which may reduce the worker's load metric and may shrink the worker's interest region. The chunk actor may then move the delegation authority of the one or more entities to one or more other worker processes. For example, referring toFIG. 10, a chunk actor (not shown) may determine to move entity940a(or other entity) from worker A to either worker B, worker C, or worker D based on a load balancing algorithm. Based on the monitoring, a chunk actor may alternatively determine whether a worker process may be at or under a minimum processing capacity. Based on the determination, the chunk actor may move all entities currently assigned to the respective worker process to another worker process based on the load balancing algorithm.

Based on the load balancing algorithm, a chunk actor may determine one or more candidate worker processes which may receive delegation authority of the one or more entities being removed from the overloaded worker process. A chunk actor may determine an initial list of candidate worker processes based on the workers which receive notifications from the overloaded worker. For example, an initial list of candidate worker processes may comprise worker processes whose interest regions overlap with the interest region of the overloaded worker. For example, workers B, C, and D may comprise an initial list of candidates for worker A, as shown inFIG. 10. A chunk actor may remove candidate worker processes from the initial list if or when a candidate worker process may be reporting a load metric that is above a predetermined threshold. Optionally, a chunk actor may be configured to determine a tensile energy for each of the one or more entities being migrated. A chunk actor may determine the tensile energy U of an entity with respect to a worker process based on a distance x (e.g.,1060a-1060c) between the entity (e.g.,940a) and the load density center1050of the respective worker and a spring constant, K (e.g., U=½ Kx2). For example, a chunk actor determining to migrate entity940amay determine a first energy between entity940aand worker B based on distance1060aand spring constant K. The chunk actor may determine a second energy between entity940aand worker C based on distance1060band spring constant K, and a third energy between entity940aand worker D based on distance1060cand spring constant K. The chunk actor may compare all of the calculated energies and determine a receiving worker which minimizes the energy between the entity and the receiving worker process. Spring constant K may be a predetermined value which may remain unchanged as the spatially-optimized simulation900progresses. In other scenarios, spring constant K may change as time progresses or based on other parameters.

In other examples, a worker process (e.g.,560, workers A, B, C, and D) may periodically relocate its interest region based on a determination of an average position of all the entities for which the worker is authoritative. For example, a worker process may move the center of its interest region (or otherwise have it moved) to the average position of all the entities for which the worker process is authoritative. Additionally, the worker process may be configured to increase or reduce in size its interest region based on its current load metric. For example, a worker process may increase a maximum simulation radius of its interest region if or when the current load metric decreases. Similarly, a worker process may decrease a maximum simulation radius of its interest region if or when the current load metric increases. In such a scenario, a chunk actor may determine a receiving worker based on the distance between the entity (e.g.,940a) and the center of the interest region of the respective worker and whether the entity is within the maximum simulation radius for the receiving worker. In yet other examples, the number and position of worker processes may remain unchanged throughout the simulation and a chunk actor may determine a receiving worker for an entity based on the location of the entity and which worker process is located closest to the entity.

A chunk actor (810,850,871,881, and891) may assign a worker process to all components of the same type comprised by the entities assigned to the chunk actor. In this manner, a worker process may simulate all the components of a certain type or all the components within a chunk region. A chunk actor (810,850,871,881, and891) may comprise multiple worker processes which may be authoritative for several entities within the chunk region.

Alternatively, the chunk actor may determine that all candidate worker processes have a load metric above the predetermined threshold. For example, a chunk actor determining to migrate entity940afrom worker A may determine that worker B, worker C, and worker D all have a load metric above the predetermined threshold. In such a scenario, the chunk actor may be configured to cause a new worker process to be instantiated and component delegation may be transferred to the newly created worker process.

A chunk actor may be further configured to utilize one of the load balancing algorithms described in detail above if or when a worker process terminates unexpectedly. For example, as described above, a worker process may cease to transmit a heartbeat signal periodically. In such a scenario, a chunk actor may migrate the entities and components which had their write authority delegated to the terminated worker process to other pre-existing worker processes. Alternatively, the chunk actor may replace the terminated worker process with a newly instantiated worker process which may have been restored using the persisted snapshot data.

Similarly, a chunk actor may be configured to utilize one of the load balancing algorithms described in detail above to assign a worker process to a newly instantiated entity. For example, entity940bmay be a newly instantiated entity and the chunk actor may utilize a load balancing algorithm to which worker process to assign the components comprised by entity940b. In the event that the chunk actor is unable to identify a worker process within its assigned region to assign to the newly instantiated entity, the chunk actor may attempt to assign the newly instantiated entity based on a local cache of known worker processes. For example, the chunk actor may maintain a local cache of known worker processes with which the chunk actor has communicated recently or within a predetermined period of time. Alternatively or additionally, the chunk actor may cause a new worker process to be instantiated and assigned to the newly instantiated entity.

Optionally, every entity in the spatially-optimized simulated world900may be configured to periodically utilize one of the load balancing algorithms described in detail above to determine whether to migrate one or more of its components to different worker process. Based on the determination and in order to effectuate the load balancing algorithm, the entity may be configured to cause a migration of the component delegation from the current worker process to another worker process.

Advantageously, and as illustrated in greater detail above, a spatially-optimized simulation development environment may automatically balance and distribute the workload across the available resources in a manner that minimizes the total amount of workers needed to perform the simulation. In addition, the spatially-optimized simulation development environment may automatically grow or shrink and move swarms of worker processes executing over possibly thousands of machines, based on the run-time workload needs of the simulation and the current location of the entities within the simulation. Furthermore, the spatially-optimized simulation development environment may dynamically recover from failures by using continuous persistence of state data and monitoring of worker process health.

FIG. 11shows a high-level architecture of an illustrative spatially-optimized simulation development environment. As shown inFIG. 11, client workers1120a-1120cmay each communicate with a bridge1140a-1140c. Similarly, server workers1130a-1130cmay each communicate with a bridge1140d-1140f. Client workers1120and server workers1130may incorporate and/or otherwise include one or more aspects of worker560as illustrated inFIGS. 5-7, and workers A, B, C, and D as illustrated inFIGS. 9-10. Client worker1120amay execute within a client computing device1110a; client worker1120bmay execute within a client computing device1110b; and, client worker1120cmay execute within a client computing device1110c. Client computing devices1110a-1110dmay incorporate and/or otherwise include one or more aspects of client computing devices340as illustrated inFIG. 3. Computing devices1110f-1110jmay comprise a server such as the server illustrated inFIGS. 2-3(e.g.,240a-240n,202a-2020, as well as other systems having different architectures (e.g. all or part ofFIG. 1.)

Bridges1140a-1140f(generally1140) may communicate with one or more chunk actors1150a-1150d(generally1150) in spatially-optimized simulation environment1100. Bridges1140may incorporate and/or otherwise include one or more aspects of bridge610as illustrated inFIGS. 6-7. Bridges1140may also communicate with each other. Chunk actors1150may incorporate and/or otherwise include one or more aspects of chunk actors810,850,871,881, and891as illustrated inFIGS. 8A-8C.

Optionally, spatially-optimized simulation environment1100may comprise a receptionist module1160. The receptionist1160may provide a well-known or predetermined network address. A client worker1120initially connecting to spatially-optimized simulation environment1100may connect to the receptionist module1160via the well-known address. The receptionist1160may receive a request to connect from a client worker1120. In response to the connection request, the receptionist1160may determine a server1110d-1110gin which to instantiate a bridge instance1140assigned to client worker1120. For example, receptionist1160may base the server determination on one of the load balancing algorithms described in detail above. In such a scenario, the receptionist1160may utilize a load balancing algorithm to assign a server1110to client worker1120. In another example, receptionist1160may maintain a coarse grain understanding of the interest region of each server1110d-1110gin the spatially-optimized simulation environment1100. In such a scenario, receptionist1160may base the server determination on the coarse grain understanding. In yet another example, each server1110d-1110gmay periodically determine an average spatial position of all bridge instances1140executing within the server1110. In such a scenario, receptionist1160may assign a server1110to client worker1120based on a comparison of the server's average spatial position with the proposed spatial position of client worker1120.

As spatially-optimized simulation1100progresses, bridge1140amay be designated to be migrated from server1110dto server1110ebased on a determination based on the load balancing algorithm described in detail above. In such a scenario, a new bridge instance1140g(not shown) may be instantiated in server1110eand client worker1120amay be temporarily connected to both bridge1140aand1140gwhile the bridge migration is effectuated. Once the migration is completed, client worker1120amay be disconnected from bridge1140aand bridge1140may be terminated. In another example, bridge1140dand server worker1130amay be designated to be migrated from server1110fto server1110g. In that scenario, a new bridge instance1140h(not shown) and a new server worker instance1130d(not shown) may be instantiated in server1110g. Server workers1130aand1130dmay be temporarily connected to bridges1140dand1140hwhile the bridge migration is effectuated. Once the migration is completed, server worker1130dmay be disconnected from bridge1140dand bridge1140dand server worker1130amay be terminated. Alternatively or additionally, bridge1140dand server worker1130amay be terminated in server1110fand restored on server1110gusing the persisted state data in snapshot326.

Optionally, spatially-optimized simulation environment1100may comprise one oracle module1170. In yet other examples, spatially-optimized simulation environment1100may comprise one oracle module1170for each virtual server cluster as described in detail above in reference toFIG. 2. An oracle module1170may comprise and maintain a workers database1172and a bridges database1174. The workers database1172may comprise data indicative of all worker instances1120and1130in a spatially-optimized simulation environment1100. Similarly, bridges database1174may comprise data indicative of all bridge instances1140in a spatially-optimized simulation environment1100. The oracle module1170may utilize the data in the workers database1172and the bridges database1174to respond to requests from a chunk actor1150for additional resources. For example, a chunk actor1150may be unable to determine a candidate worker process which may receive delegation authority. In such a scenario, chunk actor1150may request an additional worker process from oracle module1170. In response, oracle module1170may determine whether a pre-existing worker process may be available to receive the delegation authority or whether a new worker process may need to be instantiated. Based on the determination, oracle module1170may respond to chunk actor1150with data identifying a preexisting worker process. Alternatively, oracle module1170may respond to chunk actor1150with an indication that a new worker process may need to be instantiated. The oracle module1170may be further configured to utilize the data in the workers database1172and the bridges database1174to terminate worker instances and bridge instances that are underutilized or unused. For example, oracle module1170may terminate a worker instance if or when no components are assigned to the worker instance.

As previously described, workers may transmit commands to other workers, and commands may allow the worker transmitting the command to interact with an entity and/or component for which the worker is not authoritative for. Commands may comprise, for example, asynchronous remote procedure calls (RPCs), and workers may be able to send commands by invoking RPCs that may be serviced on a runtime (e.g., runtime248) server, or on another worker. In some examples, a worker may want to invoke a command addressed to some part of the system (e.g., runtime248), and/or some component and/or entity. Another worker may be authoritative over the component and/or entity. The worker desiring to invoke the command may be able to pass parameters to the command and/or receive a response indicating whether the command succeeded or failed. If the command succeeded, the worker may optionally receive a return value.

Commands may be invoked on an endpoint. An endpoint may comprise, for example, a pair of entity ID and component ID. Additionally or alternatively, an endpoint may comprise a special system endpoint for communicating with runtime248. A particular endpoint may support multiple commands. A command may be identified by a name, which may be encoded on the wire as an integer and/or may be generated by a code generator to make commands named with strings and/or generated methods. In some aspects, commands may be identified on the wire as, for example, strings, enums, namespacing, etc.

As an example, an entity might have a transform component, which may support a command called Teleport. This command might be addressed with the following address:

As another example, the runtime248may support a command to spawn an entity. This command might be address as follows:

A callee may be identified by an address (e.g., a pair of (endpoint, command name)), while a caller may be a worker and may be identified by, for example, the name of its bridge.

In some aspects, a command may be invoked successfully on another worker.FIG. 12illustrates a flow chart of a method of transmitting and/or receiving commands according to one or more illustrative aspects described herein. As previously explained, a plurality of workers associated with a computer-based, distributed simulation may be instantiated on one or more computing devices. Each worker may be authoritative over one or more entity components of a plurality of entity components associated with the computer-based simulation. As previously explained, a mapping of entity components to workers may be stored in a database associated with the simulation, such as an in-memory database. As authority changes from one worker to another worker, the mapping in the database may be updated. Also or alternatively, each worker of the plurality of workers may be associated with a bridge of a plurality of bridges. As previously explained, a worker may communicate with the simulation via its associated bridge.FIG. 12shows a worker1202associated with a bridge1204and a worker1210associated with a bridge1208. The simulation may also comprise a resolver1206. As will be described in further detail below, the resolver1206may be used to determine the bridge and/or worker to send a command to.

The worker1202(e.g., a caller of a command) may determine a command for an entity component of the simulation. If the worker1202is authoritative over the entity component, the worker1202may invoke the command on the entity component. If, on the other hand, the worker1202is not authoritative over the entity component, the worker1202may determine to send the command to another worker authoritative over the entity component, such as worker1210. The worker1202might not know the destination of the command (e.g., the identity of the bridge1208and/or worker1210). The worker1202may determine a command ID (e.g., an integer or other identifier) to identify a command invocation (e.g., cmdId inFIG. 12). The ID may be unique for the worker1202. For example, worker1202might not pick the same ID twice for two different commands. However, worker1202and worker1210could both pick the same ID. In step1232, worker1202may send, to its bridge1204, an operation, such as a request to invoke a command (e.g., InvokeCommand) on the entity component. The operation may comprise the command ID chosen by worker1202, an address of the command to invoke (e.g., an entity identifier and/or a component identifier, which may comprise an identifier for the entity component), and/or a request payload. The request payload may comprise an arbitrary protobuf. The request payload may comprise any data specified by the user. Such data may be encoded according to a schema used.

The caller's bridge (e.g., bridge1204) may receive the command invocation from worker1202. Bridge1204may determine, such as by looking up, which bridge is associated with (e.g., responsible for) the endpoint named in the address (e.g., the entity component or other endpoint). For example, in step1234, the bridge1204may transmit, to a resolver1206, a lookup request comprising the address of the endpoint (e.g., an identifier for the entity component on which to invoke the command). The resolver1206may determine, based on a mapping of entity components to workers, that the worker1210is authoritative over the entity component and/or the bridge1208associated with the worker1210. For example, the resolver1206may use the identifier for the entity component to look up, in the mapping of entity components to workers, the corresponding worker and/or the worker's bridge. In step1236, the resolver1206may respond to bridge1204indicating that bridge1208is authoritative. The response may comprise, for example, an identifier for the bridge1208. Bridge1204may use this information to determine to send the command to bridge1208. As previously explained, the mapping of entity components to workers may change, such as when authority over entity components change. Each time the worker1202desires to invoke another command on an entity component (e.g., a second command, third command, etc.), the resolver1206may use an up-to-date mapping of entity components to workers to determine the appropriate bridge and/or worker to send the command to.

In step1238, bridge1204may send a command invocation to bridge1208, and the command invocation may comprise the address, the command ID, and/or the payload request. Bridge1208may receive the request to invoke the command from bridge1204. In some examples, bridge1208may determine its own new command ID (e.g., cmdId2inFIG. 12) to talk to its worker1210. The command ID may be different from the command ID selected by the worker1202(e.g., cmdId inFIG. 12). By each bridge-worker pair using different IDs (e.g., each pair using its own sequence of IDs), the use of unique global IDs across the distributed system may be avoided. For example, the sender worker (e.g., worker1202) may generate the ID for the command request it sends, as previously explained. The sender worker1202may later match this ID to a command response received from the sender's bridge (e.g., bridge1204). The receiver's bridge (e.g., bridge1208) may generate a new ID (e.g., cmdId2inFIG. 12) for the command request the receiver's bridge1208sends to the receiver worker (e.g., worker1210). The bridge1208may match this ID to the command response received from the receiver worker1210, as will be explained in further detail below. Alternatively, bridge1208may use the same command ID as the worker1202. In step1240, the bridge1208may send, to worker1210, a request to invoke the command. The request may comprise, for example, the endpoint address (e.g., an address for the entity component), the request payload, and/or the command ID.

Worker1210may receive the invoke command request from bridge1208and may process the command, such as by invoking the command on the entity component identified in the request. In step1242, worker1210may send, to bridge1208, a response to the request to invoke the command. For example, the response may comprise a return operation. The response may comprise a command ID (e.g., the command ID sent by bridge1208to worker1210, such as cmdId2), along with a response payload (e.g., an arbitrary protobuf). The bridge1208may receive the response and may match the command ID in the response (e.g., cmdId2) to the command ID determined by the worker1202(e.g., cmdId). In step1244, the bridge1208may send a response to the bridge1204associated with the worker1202. The response sent to the bridge1204may comprise the command ID determined by the worker1202(e.g., cmdId) and/or the response payload. The bridge1204may receive the response from the bridge1208and, in step1246, may send, to the worker1202the response, which may also comprise the command ID and/or the response payload. In some examples, the response returned to the worker1202may indicate that the worker1210successfully invoked the command on the entity component. If a command response has been successfully received, then the command request may have been executed on the receiver worker once. In some other scenarios, the command may have been successfully executed or may have failed, but the receiver worker might receive a specific error or a timeout, as will be described in further detail below.

In some aspects, a command might fail on another worker, such as worker1210.FIG. 13illustrates another flow chart of a method of transmitting and/or receiving commands according to one or more illustrative aspects described herein. As previously explained, one or more of steps1232-1240may be performed to attempt to invoke a command on an entity component authoritative on the worker1210. If, however, the command fails on worker1210(e.g., the callee of the command), the worker1210may send, in step1342, a failure message back to its bridge (e.g., bridge1208). The failure message may comprise a command ID (e.g., a command ID sent by bridge1208to worker1210, such as cmdId2) and/or an explanation of the failure (e.g., the reason) back to bridge1208. A failure may include a time out for the command, such as if the worker1210took too long to respond to the command. Also or alternatively, the failure may indicate that the command was transmitted during an entity component authority handoff, where there might not be a fixed owner. Also or alternatively, the failure may comprise a user-defined failure case. Other examples of failures will be described in further detail below. The failure message may propagate back to the caller (e.g., worker1202). For example, the bridge1208may send, to the bridge1204, the response indicating that invocation of the command on the entity component failed. The bridge1204may send, to its worker1202, the response indicating that invocation of the command on the entity component failed.

Failures can occur at any stage in the above-described flows. In a first scenario, bridge1204may already have a command in flight from worker1202with the same command ID. Bridge1204may fail the command with a bad request status. In a second scenario, the resolver1206might not know about the endpoint it was asked for. For example, the entity may have been deleted. The resolver1206may fail the command with a not found status. Bridge1204may propagate the not found status back to worker1202. In a third scenario, bridge1208might not exist, and a temporary error may propagate back to worker1202. In a fourth scenario, bridge1208might not be authoritative for the invoked endpoint when the request arrives. For example, between when bridge1204asked the resolver1206where to go and the invoke message arriving at bridge1208, authority may have moved. A temporary error may be generated and propagated back to worker1202. In a fifth scenario, worker1210may have disconnected, and a temporary error may propagate back to worker1202. In a sixth scenario, any of the messages could be dropped, and a timeout may be generated at some point. In a seventh scenario, any of the messages could take a long time to be delivered, and a timeout may be generated at some point.

Any of the failures or errors described above may be propagated back to the caller (e.g., worker1202). The failure or error propagated back to the caller may comprise, for example, a status code and/or an explanation of the error. Error codes may comprise RPC error codes or other specialized error codes with well-defined semantics. The caller may make informed decisions as to whether it should retry the command or not. The intermediate components, which may be runtime248components, might not retry in response to the failure.

In some aspects, a worker (e.g., worker1202) may begin a command, but the worker may migrate to a different bridge before the response comes back to the worker. The response message from worker1210may be delivered to the new bridge (and not to the old bridge), so worker1202might see the response message. If the receiver's bridge (e.g., bridge1208) already knows about the bridge migration of the sender's bridge (e.g., bridge1204) when sending the response to the sender worker (e.g., worker1202), the receiver's bridge may send the response to the new sender bridge. If not, the sender worker (e.g., worker1202) may time out waiting for the response, which may allow the sender worker (e.g., the user code in the sender worker) to retry. In some aspects, the receiver worker (e.g., worker1210) may migrate to a new bridge. If the sender's bridge (e.g., bridge1204) already knows about the bridge migration of the receiver's bridge (e.g., bridge1208) when sending the request, the sender's bridge may send the command directly to the new receiver bridge. If not, the sender worker (e.g., worker1202) may be notified of the bridge migration, and the sender worker may re-send the request. Additionally or alternatively, the system may attempt to forward the command request to the new bridge.

In some aspects, a worker (e.g., worker1210) may begin to service a command for an entity and/or component C, but authority for entity and/or component C may move to a different worker before the command completes. In this case, the caller (e.g., worker1202) may see some kind of variant on command failed and may decide to retry or not based on the nature of the command being invoked.

In some aspects, if worker1202invokes a command, at most one worker might receive an invoke command message for the call. If worker1202receives a successful response, the command may have succeeded.

Various other scenarios may exist. If worker1202receives a response indicating that the command failed, the command may have actually succeeded. If worker1202does not receive any response at all (e.g., a timeout happens on worker1202), the command may have succeeded or failed. If worker1202invokes two commands concurrently on the same address (e.g., entity, component pair), they may return in any order. For example, worker1202may invoke command C1followed by command C2. Before command C1returns, worker1202may receive a response for command C2before receiving a response for command C1. A user code or core library may implement a built-in mechanism for attempting to provide exactly-once or idempotent command invocations. For example, a system service for generating tickets may be used.

In some examples described herein, the source of a command may be the worker that invoked the command and not a component on an entity. Workers generating the commands may provide some benefits. For example, if the source of a command is a component, workers that are not authoritative on any components or entities might not be able to invoke commands. A worker that did not invoke a command (but receives a response for the command) might be surprised. For example, some other worker who used to be authoritative on an entity that the new worker is now authoritative for may have invoked some command at some point in the past. Workers generating commands may allow recovering the same or similar semantics in user code on a per-method basis by introducing a transaction journal in the entity's states and making the relevant command idempotent (e.g., using the ticketing pattern). This is so that the command may be freely retried if authority moves between workers while a command is in flight. The newly authoritative worker may detect this condition by replaying the transaction log in the entity's state.

Various use cases for commands will now be described.

Client connection and player spawning: A player may connect to a world (e.g., a game world) by launching a client program on the player's local machine. When the client connects to the world as a worker (e.g., via the Internet), the client (e.g., a worker) may want to have a player spawned, even if the client worker does not have the permission or authority to create entities to represent the player in the world. Instead, the client worker may send a command to another worker running on the server-side to spawn an entity on behalf of the player. The worker running on the server side, which may be a player creator worker, may be authoritative for a player connection component. The player creator worker may comprise trusted code and may determine the right kind of player to be spawned (e.g., having the correct inventory, experience points, etc.).

Flammability: two (or more) workers of the same type may be authoritative over a forest of trees. Each tree may have a flammability component. When one tree sets on fire (e.g., the tree's flammability component is_on_fire becomes true), the tree may attempt to ignite other trees nearby. As there may be many workers simulating a whole forest of trees, with each other simulating a plurality of trees each, there may come a time where a tree on worker A may want to ignite a tree on worker B. By defining an ignite( ) command on the flammability component, the worker may be able to send a command to the other worker to ignite the tree, continuing the fire propagation.

Other action-at-distance simulations may use similar commands. For example, game artificial intelligence may control the behavior of a non-player character (e.g., a monster). An AI with AI components authoritative on one worker shooting another AI having its AI components authoritative on another worker.

A code example from a simulation demo introduces a concept of flammability to entities in a simulation (e.g., a game world). The simulation runtime may execute this code across a plurality of workers simultaneously, allowing flammability behavior to run across a large number of entities and to be visualized by many connected clients simultaneously.

An illustrative simulation schema is as follows:

// The simulation system schema language may let you definecross-language// components, that can be implemented and visualized across a// variety of languages and integrationscomponent Flammability {id = 1000;// This property may define a persistent state of this entitypotentially being on firebool is_on_fire = 1;// Commands may provide a way of workers interacting withother workers.// These workers may be written in different languagescommand Nothing ignite(IgniteRequest);}type IgniteRequest {// could contain additional parameters}

One or more server workers may run the following code to implement flammability:

// May be added to game objects that exist on server worker.// Potentially hundreds of server workers could be running in the same world.class PropagateFlamesBehaviour : MonoBehaviour {// FlammabilityWriter may be code generated for you// using a simulation system schema compiler, and injected[Require] private Flammability.Writer flammabilityWriter;void OnEnable( ) {// Set up a handler for when we get an Ignite commandflammabilityWriter.CommandReceiver.OnIgnite.RegisterResponse(HandleIgnite);}// When the trigger around the flaming game object intersects another entityvoid OnTriggerEnter(Collider other) {if (flammabilityWriter.Data.isOnFire) {// Simulation commands may route to whichever worker issimulating// this component.Simulation.Commands.SendCommand(flammabilityWriter,Flammability.Commands.Ignite.Descriptor,new IgniteRequest( ),other.gameObject.EntityId( ));}}private Nothing HandleIgnite(Nothing request, ICommandCallerInfo callerinfo) {if (!flammabilityWriter.Data.isOnFire) {flammabilityWriter.Send(new Flammable.Update( ).SetIsOnFire(true));}return new Nothing( );}void OnDisable( ) {flammabilityWriter.CommandReceiver.OnIgnite.DeregisterResponse( );}}

One or more client workers may run the following code to implement flammability:

// May be added to game objects that exist on client worker.// Arbitrarily many clients may be running this codeclass FlamesVisualizer : MonoBehaviour {// FlammabilityReader may be code generated for you// using simulation system schema compiler, and injected[Require] private Flammability.Reader flammabilityReader;// This may reference a particle system on the game objectpublic ParticleSystem particles;void OnEnable( ) {// Re-visualize when we are set on fireflammabilityReader.IsOnFireUpdated.Add(UpdateVisualization);UpdateVisualization(flammabilityReader.Data.isOnFire);}void UpdateVisualization(bool isOnFire) {// Enable the particle system as appropriateif (isOnFire) {particles.Start( );} else {particles.Stop( );}}}

Whilst the embodiments and aspects have been described in relation to virtual hardware servers, the methods and systems may also be used with other hardware or servers including local or physical servers.