Package: ltron
Interactive Lego Machine Learning EnvironmentLTRON is an environment for interactive machine learning assembly problems using Lego bricks.  It is based onLDrawtheOpen Model Repository, and uses additional data fromLDCAD.If you use LTRON in acadmic work, please cite ourECCV 2022 Paper:@inproceedings{walsman2022break,
  author = {Aaron Walsman, Muru Zhang, Klemen Kotar, Karthik Desingh, Ali Farhadi, Dieter Fox},
  title = {Break and Make: Interactive Structural Understanding Using LEGO Bricks},
  booktitle = {European Conference on Computer Vision},
  year={2022}
}Examples8661-1 - Carbon Star.mpd31025 - Mountain Hut - Mountain Hut.mpd10030-1 - Imperial Star Destroyer - UCS.mpdInstallation:To install LTRON run:pip install ltron
ltron_asset_installerThe first line will download LTRON from pypi and install it in your python path. The second line will download and install a set of models and part files that are necessary for LTRON.  By default, these assets are installed to~/.cache/ltronand~/.cache/splendor.  This will take around 3GB of space.Release Versions:Embodied AI Workshop, CVPR 2021:(pypi 0.0.X) (branch TODO).ECCV 2022:(pypi 1.0.X) (branch v1.0.0)Ongoing Work (not in pypi) (branch master)Testing Things Out:Once installed you should be able to runltron_viewerto interactively inspect a lego model.  For example from the top directory you can run:ltron_viewer "~/.cache/ltron/collections/omr/ldraw/8661-1 - Carbon Star.mpd"Or:ltron_viewer "~/.cache/ltron/collections/omr/ldraw/75050-1 - B-Wing.mpd"Or:ltron_viewer "~/.cache/ltron/collections/omr/ldraw/75060 - Slave I.mpd"You can interact with the model by clicking on it and dragging the mouse around.  LMB - Orbit.  RMB - Pan.  Scroll - Zoom.  There are a few keys you can presshto hide the brick you are hovering over,vto show all hidden bricks andmto switch back and forth between mask mode and regular rendering.  See other options inltron/visualization/ltron_viewer.py.Requirements:gym
numpy
pyquaternion
gdown
tqdm
splendor-renderThe splendor-render package only works on Ubuntu at the moment, and requires OpenGL 4.6.  As long as you have a modern GPU with recent drivers you should be fine.You may need to install freeglut:sudo apt-get install freeglut3-devFile FormatsLTRON uses theLDraw file formats(.dat, .ldr, .mpd) to describe bricks and models.  We convert all bricks to obj files for use in splendor-render.  These are installed to~/.cache/splendor/ltron_assets_lowor~/.cache/splendor/ltron_assets_high.The Open Model Repository files are installed to~/.cache/ltron/collections/omr/ldraw.Data Layout:This section is current as of v1.0.X, but may be streamlined in future releases:As noted above, the LTRON assets are placed by default in~/.cache/ltron.  This location can be changed by setting the$LTRON_HOMEenvironment variable.  Additionally, some rendering assets are placed in~/.cache/splendor.  This location can be changed by setting the $SPLENDOR_HOME` environment variable.All LTRON datasets are registered in$LTRON_HOME/settings.cfg.  In LTRON each dataset contains a json file that describes where to find the files associated with that dataset, and various metadata.  These are listed in the[datasets]header in thesettings.cfgfile.  These json files usually refer to a particular "collection" of ldraw files.  You can think of a collection as a root file path where several ldraw files and/or episode zip files live.  These locations are specified in the[collections]header in thesettings.cfgfile.Looking the$LTRON_HOME/collections/random_construction_6_6/rc_6_6.jsonfile we can see what kind of data it contains:splits: a set of names corresponding to a block of files used for training or testing.rc_6_6.jsoncontains nine splits: atrain_N,test_Nandtrain_episodes_Nfor N=2,4,8.  The path to each of these splits starts with{random_construction_6_6}/...which tells the system to look for these files inside therandom_construction_6_6collection.max_instances_per_scene: an upper bound on number of instances that exist in any of the ldraw files used in the dataset.max_edges_per_scene: an upper bound on the number of connections between bricks that exist in any of the ldraw files used in this dataset.shape_ids: a class label for each brick shape used in this datasetclass_ids: a separate class label for each brick color used in this datasetUsing this structure provides access to the following commands:ltron.dataset.get_dataset_info('dataset_name')returns the contents of the json dataset fileltron.dataset.get_dataset_paths('dataset_name', 'split_name', subset=None, rank=0, size=1)returns a list of paths belonging to a particular split.ltron.dataset.get_zip_paths('dataset_name', 'split_name', subset=None, rank=0, size=1)returns a zipfile object and a list of its contents for a particular split.LtronEnv StructureWe have tried to make the LtronEnv structure modular so that it can be customized for a variety of observation types, action spaces and tasks.  Therefore the LtronEnvgymenvironment is a container for multiple components, each of which specify their own action/observation spaces and state variables.  When LtronEnv's reset/step/etc. method is called, it executes each of its component's reset/step/etc. methods in order and accumulating the result into a dictionary based on the component's name.  Currently only the Break and Make (ltron/gym/envs/break_and_make_env.py in v1.0.0) is stable.Further Notes:All 3D units are inLDraw LDUs.  One LDU is approximately 0.4 mm, so the physical extents of these scenes can be quite large.