Package: inferactively-pymdp
A Python package for simulating Active Inference agents in Markov Decision Process environments.
Please see our companion paper, published in the Journal of Open Source Software:"pymdp: A Python library for active inference in discrete state spaces"for an overview of the package and its motivation. For a more in-depth, tutorial-style introduction to the package and a mathematical overview of active inference in Markov Decision Processes, see thelonger arxiv versionof the paper.This package is hosted on theinfer-activelyGitHub organization, which was built with the intention of hosting open-source active inference and free-energy-principle related software.Most of the low-level mathematical operations areNumPyports of their equivalent functions from theSPMimplementationin MATLAB. We have benchmarked and validated most of these functions against their SPM counterparts.Statuspymdpin actionHere's a visualization ofpymdpagents in action. One of the defining features of active inference agents is the drive to maximize "epistemic value" (i.e. curiosity). Equipped with such a drive in environments with uncertain yet disclosable hidden structure, active inference can ultimately allow agents to simultaneously learn about the environment as well as maximize reward.The simulation below (see associated notebookhere) demonstrates what might be called "epistemic chaining," where an agent (here, analogized to a mouse seeking food) forages for a chain of cues, each of which discloses the location of the subsequent cue in the chain. The final cue (here, "Cue 2") reveals the location a hidden reward. This is similar in spirit to "behavior chaining" used in operant conditioning, except that here, each successive action in the behavioral sequence doesn't need to be learned through instrumental conditioning. Rather, active inference agents will naturally forage the sequence of cues based on an intrinsic desire to disclose information. This ultimately leads the agent to the hidden reward source in the fewest number of moves as possible.You can run the code behind simulating tasks like this one and others in theExamplessection of theofficial documentation.Cue 2 in Location 1, Reward on TopCue 2 in Location 3, Reward on BottomQuick-start: Installation and UsageIn order to usepymdpto build and develop active inference agents, we recommend installing it with the the package installerpip, which will installpymdplocally as well as its dependencies. This can also be done in a virtual environment (e.g. withvenv).When pip installingpymdp, use the package nameinferactively-pymdp:pipinstallinferactively-pymdpOnce in Python, you can then directly importpymdp, its sub-packages, and functions.importpymdp
frompymdpimportutils
frompymdp.agentimportAgentnum_obs=[3,5]# observation modality dimensionsnum_states=[3,2,2]# hidden state factor dimensionsnum_controls=[3,1,1]# control state factor dimensionsA_matrix=utils.random_A_matrix(num_obs,num_states)# create sensory likelihood (A matrix)B_matrix=utils.random_B_matrix(num_states,num_controls)# create transition likelihood (B matrix)C_vector=utils.obj_array_uniform(num_obs)# uniform preferences# instantiate a quick agent using your A, B and C arraysmy_agent=Agent(A=A_matrix,B=B_matrix,C=C_vector)# give the agent a random observation and get the optimized posterior beliefsobservation=[1,4]# a list specifying the indices of the observation, for each observation modalityqs=my_agent.infer_states(observation)# get posterior over hidden states (a multi-factor belief)# Do active inferenceq_pi,neg_efe=my_agent.infer_policies()# return the policy posterior and return (negative) expected free energies of each policy as wellaction=my_agent.sample_action()# sample an action# ... and so on ...Getting started / introductory materialWe recommend starting with the Installation/Usage section of theofficial documentationfor the repository, which provides a series of useful pedagogical notebooks for introducing you to active inference and how to build agents inpymdp.For new users topymdp, we specifically recommend stepping through following three Jupyter notebooks (can also be used on Google Colab):PymdpfundamentalsActive Inference from ScratchTheAgentAPISpecial thanks toBeren MillidgeandDaphne Demekasfor their help in prototyping earlier versions of theActive Inference from Scratchtutorial, which were originally based on a grid world POMDP environment create byAlec Tschantz.We also have (and are continuing to build) a series of notebooks that walk through active inference agents performing different types of tasks, such as the classicT-Maze environmentand the newerEpistemic Chainingdemo.ContributingThis package is under active development. If you would like to contribute, please refer tothis fileIf you would like to contribute to this repo, we recommend using venv and pipcd<path_to_repo_fork>
python3-mvenvenvsourceenv/bin/activate
pipinstall-rrequirements.txt
pipinstall-e./# This will install pymdp as a local dev packageYou should then be able to run tests locally withpytestpytesttestCitingpymdpIf you usepymdpin your work or research, please consider citing ourpaper(open-access) published in the Journal of Open-Source Software:@article{Heins2022,
  doi = {10.21105/joss.04098},
  url = {https://doi.org/10.21105/joss.04098},
  year = {2022},
  publisher = {The Open Journal},
  volume = {7},
  number = {73},
  pages = {4098},
  author = {Conor Heins and Beren Millidge and Daphne Demekas and Brennan Klein and Karl Friston and Iain D. Couzin and Alexander Tschantz},
  title = {pymdp: A Python library for active inference in discrete state spaces},
  journal = {Journal of Open Source Software}
}For a more in-depth, tutorial-style introduction to the package and a mathematical overview of active inference in Markov Decision Processes, you can also consult thelonger arxiv versionof the paper.AuthorsConor Heins@conorheinsAlec Tschantz@alec-tschantzBeren Millidge@BerenMillidgeBrennan Klein@jkbrenArun Niranjan@Arun-NiranjanDaphne Demekas@daphnedemekas