Package: minedojo
Building Open-Ended Embodied Agents with Internet-Scale Knowledge[Website][Arxiv Paper][PDF][Docs][Open Database][Team]is a new AI research framework for building open-ended, generally capable embodied agents.
MineDojo features amassive simulation suitebuilt on Minecraft with 1000s of diverse tasks, and providesopen access to an internet-scale knowledge baseof 730K YouTube videos, 7K Wiki pages, 340K Reddit posts.Using MineDojo, AI agents can freely explore a procedurally generated 3D world with diverse terrains to roam :earth_asia:	, materials to mine :gem:, tools to craft :wrench:, structures to build :european_castle:, and wonders to discover :sparkles:. Instead of training in isolation, your agent will be able to learn from the collective wisdom of millions of human players around the world!InstallationGetting StartedBenchmarking SuiteProgrammatic TasksCreative TasksPlaythrough TaskOur PaperLicenseInstallationMineDojo requires Python ≥ 3.9. We have tested on Ubuntu 20.04 and Mac OS X.Please followthis guideto install the prerequisites first, such as JDK 8 for running Minecraft backend. We highly recommend creating a newConda virtual envto isolate dependencies. Installing the MineDojo stable version is as simple as:pipinstallminedojoTo install the cutting edge version from the main branch of this repo, run:gitclonehttps://github.com/MineDojo/MineDojo&&cdMineDojo
pipinstall-e.You can run the script below to verify the installation. It takes a while to compile the Java code for the first time. After that you should see a Minecraft window pop up, with the same gaming interface that human players receive. You should see the message[INFO] Installation Successif everything goes well.pythonminedojo/scripts/validate_install.pyNote that if you are on a headless machine, don't forget to prepend eitherxvfb-runorMINEDOJO_HEADLESS=1:xvfb-runpythonminedojo/scripts/validate_install.py# --- OR ---MINEDOJO_HEADLESS=1pythonminedojo/scripts/validate_install.pyGetting StartedMineDojo provides aGym-styleinterface for developing embodied agents that interact with the simulator in a loop.  Here is a very simple code snippet of a hardcoded agent that runs forward and jumps every 10 steps in the "Harvest Wool" task:importminedojoenv=minedojo.make(task_id="harvest_wool_with_shears_and_sheep",image_size=(160,256))obs=env.reset()foriinrange(50):act=env.action_space.no_op()act[0]=1# forward/backwardifi%10==0:act[2]=1# jumpobs,reward,done,info=env.step(act)env.close()Please refer tothis tutorialfor a detailed walkthrough of your first agent. MineDojo features a multimodal observation space (RGB, compass, voxels, etc.) and a compound action space (movement, camera, attack, craft, etc.). Seethis docto learn more. We recommend you to reference the fullobservationandaction spacespecifications.MineDojo can be extensively customized to be tailored to your research needs. Please check out customization guides ontasks,simulation, andprivileged observation.Benchmarking SuiteMineDojo features a massively multitask benchmark with3131 tasksin the current release.
We design a unified top-level functionminedojo.make(), similar togym.make, that creates all the tasks and environments in our benchmarking suite. We categorize the tasks intoProgrammatic,Creative, andPlaythrough.Task CategoryCountDescriptionProgrammatic1572Can be automatically scored based on ground-truth simulator statesCreative1558Do not have well-defined or easily-automated success criteriaPlaythrough1Special achievement: defeat the Ender dragon, "beat the game"We pair all tasks with natural language descriptions of task goals (i.e. "prompts"), such as"obtain 8 bone in swampland"and"make a football stadium". Many tasks also have step-by-step guidance generated by GPT-3. Users can access a comprehensive listing of prompts and guidance for all task by:# list of string IDsall_ids=minedojo.tasks.ALL_TASK_IDS# dict: {task_id: (prompt, guidance)}all_instructions=minedojo.tasks.ALL_TASK_INSTRUCTIONSProgrammatic Tasks1572 Programmatic tasks can be further divided into four categories: (1)Survival: surviving for a designated number of days, (2)Harvest: finding, obtaining, cultivating, or manufacturing hundreds of materials and objects, (3)Tech Tree: the skills of crafting and using a hierarchy of tools, and (4)Combat:  fight various monsters and creatures to test agent's reflex and martial skills. Refer tothis docfor more information.The following code creates a Programmatic task with IDharvest_milkwith 160x256 resolution:env=minedojo.make(task_id="harvest_milk",image_size=(160,256))You can access task-related attributes such astask_promptandtask_guidance:>>>env.task_prompt
obtainmilkfromacow
>>>env.task_guidance1.Findacow.2.Right-clickthecowwithanemptybucket.Here we show a few examples from each category:Task PromptVisualizationTask PromptVisualizationshear a sheep with shears and a sheep nearbyobtain milk from a cows in forest with an empty bucketobtain 8 ghast tearobtain chicken in swamplandcombat a husk in night desert with a diamond sword, shield, and a full suite of iron armorshunt a bat in night plains with a iron sword, shield, and a full suite of diamond armorscombat a spider in night forest with a wooden sword, shield, and a full suite of iron armorshunt a pig in extreme hills with a wooden sword, shield, and a full suite of leather armorsstarting from wood tools, craft and use a diamond swordstarting from stone tools, craft and use a tntstarting from gold tools, craft and use a clockstarting from diamond tools, craft and use a dispensersurvive as long as possiblesurvive as long as possible given a sword and some foodCreative TasksSimilar to Programmatic tasks, Creative tasks can be instantiated byminedojo.make(). The only difference is thattask_idno longer has any semantic meaning. Instead, the format becomsecreative:{task_index}. You can query all Creative task IDs fromminedojo.tasks.ALL_CREATIVE_TASK_IDS.The following code instantiates the 256th task from our Creative suite:env=minedojo.make(task_id="creative:255",image_size=(160,256))Let's see what the task prompt and guidance are:>>>env.task_prompt
BuildareplicaoftheGreatPyramidofGiza
>>>env.task_guidance1.Findadesertbiome.2.Findaspotthatis64blockswideand64blockslong.3.Makeafoundationthatis4blockshigh.4.Makethefirstlayerofthepyramidusingblocksthatare4blockswideand4blockslong.5.Makethesecondlayerofthepyramidusingblocksthatare3blockswideand3blockslong.6.Makethethirdlayerofthepyramidusingblocksthatare2blockswideand2blockslong.7.Makethefourthlayerofthepyramidusingblocksthatare1blockwideand1blocklong.8.Makethecapstoneofthepyramidusingablockthatis1blockwideand1blocklong.Please refer tothis docfor more details on Creative tasks.Playthrough TaskPlaythrough task's instruction is to "Defeat the Ender Dragon and obtain the trophy dragon egg". This task holds a unique position because killing the dragon means "beating the game" in the traditional sense of the phrase, and is considered the most significant achievement for a new player. The mission requires lots of preparation, exploration, agility, and trial-and-error, which makes it a grand challenge for AI:env=minedojo.make(task_id="playthrough",image_size=(160,256))Using the Knowledge BaseMinecraft has more than 100M active players, who have collectively generated an enormous wealth of data.MineDojo features a massive database collected automatically from the internet.AI agents can learn from this treasure trove of knowledge to harvest actionable insights, acquire diverse skills, develop complex strategies, and discover interesting objectives to pursue. All our databases areopen-access and available to download today!YouTube DatabaseMinecraft is among the most streamed games on YouTube. Human players have demonstrated a stunning range of creative activities and sophisticated missions that take hours to complete.We collect 730K+ narrated Minecraft videos, which add up to ~300K hours and 2.2B words in English transcripts.The time-aligned transcripts enable the agent to ground free-form natural language in video pixels and learn the semantics of diverse activities without laborious human labeling. Please refer to thedoc pagefor how to load our YouTube database.Wiki DatabaseThe Wiki pages cover almost every aspect of the game mechanics, and supply a rich source of unstructured knowledge in multimodal tables, recipes, illustrations, and step-by-step tutorials.We scrape ~7K pages that interleave text, images, tables, and diagrams.To preserve the layout information, we also save the screenshots of entire pages and extract bounding boxes of the visual elements. Please refer to thedoc pagefor how to load our Wiki database.Reddit DatabaseWe collect 340K+ Reddit posts along with 6.6M comments under the “r/Minecraft” subreddit.These posts ask questions on how to solve certain tasks, showcase cool architectures and achievements in image/video snippets, and discuss general tips and tricks for players of all expertise levels. Large language models can be finetuned on our Reddit corpus to internalize Minecraft-specific concepts and develop sophisticated strategies. Please refer to thedoc pagefor how to load our Reddit database.Check out our paper!Our paper is available onArxiv. If you find our code or databases useful, please consider citing us!@article{fan2022minedojo,title={MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge},author={Linxi Fan and Guanzhi Wang and Yunfan Jiang and Ajay Mandlekar and Yuncong Yang and Haoyi Zhu and Andrew Tang and De-An Huang and Yuke Zhu and Anima Anandkumar},year={2022},journal={arXiv preprint arXiv: Arxiv-2206.08853}}LicenseComponentLicenseCodebase (this repo)MIT LicenseYouTube DatabaseCreative Commons Attribution 4.0 International (CC BY 4.0)Wiki DatabaseCreative Commons Attribution Non Commercial Share Alike 3.0 UnportedReddit DatabaseCreative Commons Attribution 4.0 International (CC BY 4.0)