Package: maze-dataset
maze-datasetThis package provides utilities for generation, filtering, solving, visualizing, and processing of mazes for training ML systems. Primarily built for themaze-transformer interpretabilityproject. You can find our paper on it here:http://arxiv.org/abs/2309.10498This package includes a variety ofmaze generation algorithms, including randomized depth first search, Wilson's algorithm for uniform spanning trees, and percolation. Datasets can be filtered to select mazes of a certain length or complexity, remove duplicates, and satisfy custom properties. A variety of output formats for visualization and training ML models are provided.UsageMost of the functionality is demonstrated in the ipython notebooks in thenotebooks/folder.demo_dataset.ipynbhow to easily create a dataset of mazes, utilities for filtering the generates mazes via properties, and basic visualization. View this one first.demo_tokenization.ipynbconverting mazes to and from textual representations, as well as utilities for working with them.demo_latticemaze.ipynbinternals of theLatticeMazeandSolvedMazeobjects, and advanced visualization.Creating a datasetTo create aMazeDataset, which inherits fromtorch.utils.data.Dataset, you first create aMazeDatasetConfig:frommaze_datasetimportMazeDataset,MazeDatasetConfigfrommaze_dataset.generationimportLatticeMazeGeneratorscfg:MazeDatasetConfig=MazeDatasetConfig(name="test",# name is only for you to keep track of thingsgrid_n=5,# number of rows/columns in the latticen_mazes=4,# number of mazes to generatemaze_ctor=LatticeMazeGenerators.gen_dfs,# algorithm to generate the mazemaze_ctor_kwargs=dict(do_forks=False),# additional parameters to pass to the maze generation algorithm)and then pass this config to theMazeDataset.from_configfactory method:dataset:MazeDataset=MazeDataset.from_config(# your configcfg,# and all this below is completely optional# do_download=False,# load_local=False,# do_generate=True,# save_local=True,# gen_parallel=False,)This method can search for whether a dataset with matching config hash already exists on your filesystem in the expected location, and load it if so. It can also generate a dataset on the fly if needed.Conversions to useful formatsThe elements of the dataset areSolvedMazeobjects:>>>m=dataset[0]>>>type(m)maze_dataset.maze.lattice_maze.SolvedMazeWhich can be converted to a variety of formats:# visual representation as ascii artm.as_ascii()# RGB image, optionally without solution or endpoints, suitable for CNNsm.as_pixels()# text format for autoreregressive transformersfrommaze_dataset.tokenizationimportMazeTokenizer,TokenizationModem.as_tokens(maze_tokenizer=MazeTokenizer(tokenization_mode=TokenizationMode.AOTP_UT_rasterized,max_grid_size=100,))# advanced visualization with many featuresfrommaze_dataset.plottingimportMazePlotMazePlot(maze).plot()InstallationThis package isavailable on PyPI, and can be installed viapip install maze-datasetDevelopmentThis project usesPoetryfor development. To install with dev requirements, runpoetry install --with devA makefile is included to simplify common development tasks:make helpwill print all available commandsall tests viamake testunit tests viamake unitnotebook tests viamake test_notebooksformatter (black, pycln, and isort) viamake formatformatter in check-only mode viamake check-formatCitingIf you use this code in your research, please citeour paper:@misc{maze-dataset,
    title={A Configurable Library for Generating and Manipulating Maze Datasets}, 
    author={Michael Igorevich Ivanitskiy and Rusheb Shah and Alex F. Spies and Tilman Räuker and Dan Valentine and Can Rager and Lucia Quirke and Chris Mathwin and Guillaume Corlouer and Cecilia Diniz Behn and Samy Wu Fung},
    year={2023},
    eprint={2309.10498},
    archivePrefix={arXiv},
    primaryClass={cs.LG},
    url={http://arxiv.org/abs/2309.10498}
}