Package: nett-benchmarks
Newborn Embodied Turing TestBenchmarking Virtual Agents in Controlled-Rearing ConditionsGetting Started•Documentation•Lab WebsiteThe Newborn Embodied Turing Test (NETT) is a cutting-edge toolkit designed to simulate virtual agents in controlled-rearing conditions. This innovative platform enables researchers to create, simulate, and analyze virtual agents, facilitating direct comparisons with real chicks as documented by theBuilding a Mind Lab. Our comprehensive suite includes all necessary components for the simulation and analysis of embodied models, closely replicating laboratory conditions.Below is a visual representation of our experimental setup, showcasing the infrastructure for the three primary experiments discussed in this documentation.How to Use this RepositoryThe NETT toolkit comprises three key components:Virtual Environment: A dynamic environment that serves as the habitat for virtual agents.Experimental Simulation Programs: Tools to initiate and conduct experiments within the virtual world.Data Visualization Programs: Utilities for analyzing and visualizing experiment outcomes.Directory StructureThe directory structure of the code is as follows:├── docs                       # Documentation and guides
├── notebooks
│   ├── Getting Started.ipynb  # Introduction and setup notebook
├── src/nett
│   ├── analysis               # Analysis scripts
│   ├── body                   # Agent body configurations
│   ├── brain                  # Neural network models and learning algorithms
│   ├── environment            # Simulation environments
│   ├── utils                  # Utility functions
│   ├── nett.py                # Main library script
│   └── __init__.py            # Package initialization
├── tests                      # Unit tests
├── mkdocs.yml                 # MkDocs configuration
├── pyproject.toml             # Project metadata
└── README.md                  # This README fileGetting StartedTo begin benchmarking your first embodied agent with NETT, please be aware:Important: Themlagents==1.0.0dependency is incompatible with Apple Silicon (M1, M2, etc.) chips. Please utilize an alternate device to execute this codebase.InstallationVirtual Environment Setup(Highly Recommended): Create and activate a virtual environment to avoid dependency conflicts.condacreate-y-nnett_envpython=3.10.12
condaactivatenett_envSeeherefor detailed instructions.Install the needed versions ofsetuptoolsandpip:pipinstallsetuptools==65.5.0pip==21wheel==0.38.4NOTE:This is a result of incompatibilities with the subdependencygym==0.21. More information about this issue can be foundhereToolkit Installation: Install the toolkit usingpip.pipinstallnett-benchmarksNOTE:: Installation outside a virtual environment may fail due to conflicting dependencies. Ensure compatibility, especially withgym==0.21andnumpy<=1.21.2.Running a NETTDownload or Create the Unity Executable: Obtain a pre-made Unity executable fromhere. The executable is required to run the virtual environment.Import NETT Components:
Start by importing the NETT framework components -Brain,Body, andEnvironment, alongside the mainNETTclass.fromnettimportBrain,Body,EnvironmentfromnettimportNETTComponent Configuration:Brain: Configure the learning aspects, including the policy network (e.g. "CnnPolicy"), learning algorithm (e.g. "PPO"), the reward function, and the encoder.brain=Brain(policy="CnnPolicy",algorithm="PPO")To get a list of all available policies, algorithms, and encoders, theBrainclass contains the methodslist_policies(),list_algorithms(), andlist_encoders()respectively.Body: Set up the agent's physical interface with the environment. It's possible to apply gym.Wrappers for data preprocessing.body=Body(type="basic",dvs=False,wrappers=None)Here, we do not pass any wrappers, letting information from the environment reach the brain "as is". Alternative body types (e.g.two-eyed,rag-doll) are planned in future updates.Environment: Create the simulation environment using the path to your Unity executable (see Step 1).environment=Environment(config="identityandview",executable_path="path/to/executable.x86_64")To get a list of all available configurations, runEnvironment.list_configs().Run the Benchmarking:  Integrate all components into a NETT instance to facilitate experiment execution.benchmarks=NETT(brain=brain,body=body,environment=environment)TheNETTinstance has a.run()method that initiates the benchmarking process. The method accepts parameters such as the number of brains, training/testing episodes, and the output directory.job_sheet=benchmarks.run(output_dir="path/to/run/output/directory/",num_brains=5,trains_eps=10,test_eps=5)Therunfunction is asynchronous, returning the list of jobs that may or may not be complete.Check Status: To see the status of the benchmark processes, use the.status()method:benchmarks.status(job_sheet)Running Standard AnalysisAfter running the experiments, the pipeline will generate a collection of datafiles in the defined output directory.Install R and dependencies: To run the analyses performed in previous experiments,this toolkit provides a set of analysis scripts. Prior to running them, you will need R and the packagestidyverse,argparse, andscalesinstalled. To install these packages, run the following command in R:install.packages(c("tidyverse","argparse","scales"))Alternatively, if you are having difficulty installing R on your system, you can install these using conda.condainstall-yrr-tidyverser-argparser-scalesRun the Analysis: To run the analysis, use theanalyzemethod of theNETTclass. This method will generate a set of plots and tables based on the datafiles in the output directory.benchmarks.analyze(run_dir="path/to/run/output/directory/",output_dir="path/to/analysis/output/directory/")DocumentationFor a link to the full documentation, please visithere.Experiment ConfigurationMore information related to details on the experiment can be found on following pages.Parsing ExperimentViewPoint Experiment🔼 Back to top