## 2 Materials and Methods
### System Design
#### 2.1.5 System Distributability  
The multi-agent nature of this system lends itself well to being distributed across multiple computational nodes using a cluster of otherwise independent machines. There are three main phases of this work to consider for distributability, the per-agent training phase where each small MLP network is trained, the social interaction phase, and the swarming phase where agents interact and vote before a system-wide prediction is made. The training phase relies on a small amount of data which can be made available on a node-by-node basis or transferred across the network without significant delay. During this phase each agent is independent and distribution is straightforward. Agents can be placed on each node randomly, or using an initialization algorithm to optimally pick a node for each agent based on computational load or network limitations. The MLP network for each agent can be trained independently from any other agent and the small size and depth of the network do not limit training to powerful hardware. The social interaction phase initializes all participating agents in the interaction arena. The flexible design of the arena allows each compute node to represent some subdivision of the full arena, e.g., an arena with 4 nodes and 120 possible spaces will be subdivided so that each node is responsible for a block of 30 spaces. An agent moving between each subdivision is moved between nodes. The movement algorithm governing each agent can encourage or discourage movement between nodes to balance information dispersal with network transit times. Figure 1 shows a possible configuration with a single interaction arena split across 16 nodes. The dark outline represents the bounds of the arena and the lighter internal lines represent boundaries between nodes. Agents can move between nodesfreely during the interaction period, which requires network transfers, either explicitly from some node to another node, or implicitly through shared storage. The work presented here takes advantage of a disk-based database and a networked Ceph storage cluster that is shared by each node to implicitly share data on each node without explicit network transfer; no agent is serialized for transfer across the network stack. After each iteration in the interaction arena, the state of each agent is flushed to an SQLite database, which is updated and accessible to all nodes using the networked Ceph storage cluster. As each agent crosses a boundary between nodes, a per-agent flag is set indicating on which node the agent is present. Each node checks a table to determine which agents are present on that node prior to the start of every interaction iteration. The swarming phase of the system can require frequent movement of agents between nodes due to the frequent re-selection process for \"presenters\" and \"watchers\" while the swarm is iterating. The selection process randomly selects \"presenters\" and then distributes agents to available nodes. This can reduce the benefit of distributing this method across multiple compute nodes, due to the additional run time cost from network transfers. The swarming phase of the system can work in two distinct modes of operation. The first option requires a simple command and control node, which is responsible for selecting the \"presenters\" and \"watchers\", as described in Section 2.1.2, and uniformly distributes the agents to available nodes. Each additional iteration of the swarming process does, however, require global knowledge and control to re-distribute agents to each node. This method does produce results identical to swarming on a single node, but is the slower of the two options. The second option allows localized swarming on each node with aggregation at the end of the swarming process. Following the interaction phase, the agents can either stay on their current node, or agents can be randomly distributed to nodes. There is no additional movement of agents between nodes throughout the swarming process.