Package: ravens
Ravens - Transporter NetworksRavens is a collection of simulated tasks in PyBullet for learning vision-based robotic manipulation, with emphasis on pick and place.
It features a Gym-like API with 10 tabletop rearrangement tasks, each with (i) a scripted oracle that provides expert demonstrations (for imitation learning), and (ii) reward functions that provide partial credit (for reinforcement learning).(a)block-insertion: pick up the L-shaped red block and place it into the L-shaped fixture.(b)place-red-in-green: pick up the red blocks and place them into the green bowls amidst other objects.(c)towers-of-hanoi: sequentially move disks from one tower to another—only smaller disks can be on top of larger ones.(d)align-box-corner: pick up the randomly sized box and align one of its corners to the L-shaped marker on the tabletop.(e)stack-block-pyramid: sequentially stack 6 blocks into a pyramid of 3-2-1 with rainbow colored ordering.(f)palletizing-boxes: pick up homogeneous fixed-sized boxes and stack them in transposed layers on the pallet.(g)assembling-kits: pick up different objects and arrange them on a board marked with corresponding silhouettes.(h)packing-boxes: pick up randomly sized boxes and place them tightly into a container.(i)manipulating-rope: rearrange a deformable rope such that it connects the two endpoints of a 3-sided square.(j)sweeping-piles: push piles of small objects into a target goal zone marked on the tabletop.Some tasks require generalizing to unseen objects (d,g,h), or multi-step sequencing with closed-loop feedback (c,e,f,h,i,j).Team:this repository is developed and maintained byAndy Zeng,Pete Florence,Daniel Seita,Jonathan Tompson, andAyzaan Wahid. This is the reference repository for the paper:Transporter Networks: Rearranging the Visual World for Robotic ManipulationProject Website•PDF•  Conference on Robot Learning (CoRL) 2020Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong,Ivan Krasin, Dan Duong, Vikas Sindhwani, Johnny LeeAbstract.Robotic manipulation can be formulated as inducing a sequence of spatial displacements: where the space being moved can encompass an object, part of an object, or end effector. In this work, we propose the Transporter Network, a simple model architecture that rearranges deep features to infer spatial displacements from visual input—which can parameterize robot actions. It makes no assumptions of objectness (e.g. canonical poses, models, or keypoints), it exploits spatial symmetries, and is orders of magnitude more sample efficient than our benchmarked alternatives in learning vision-based manipulation tasks: from stacking a pyramid of blocks, to assembling kits with unseen objects; from manipulating deformable ropes, to pushing piles of small objects with closed-loop feedback. Our method can represent complex multi-modal policy distributions and generalizes to multi-step sequential tasks, as well as 6DoF pick-and-place. Experiments on 10 simulated tasks show that it learns faster and generalizes better than a variety of end-to-end baselines, including policies that use ground-truth object poses. We validate our methods with hardware in the real world.InstallationStep 1.Recommended: installMinicondawith Python 3.7.curl-Ohttps://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bashMiniconda3-latest-Linux-x86_64.sh-b-uecho$'\nexport PATH=~/miniconda3/bin:"${PATH}"\n'>>~/.profile# Add Conda to PATH.source~/.profile
condainitStep 2.Create and activate Conda environment, then install GCC and Python packages.cd~/ravens
condacreate--nameravenspython=3.7-y
condaactivateravens
sudoapt-getupdate
sudoapt-get-yinstallgcclibgl1-mesa-dev
pipinstall-rrequirements.txt
pythonsetup.pyinstall--userStep 3.Recommended: install GPU acceleration with NVIDIACUDA10.1 andcuDNN7.6.5 for Tensorflow../oss_scipts/install_cuda.sh#  For Ubuntu 16.04 and 18.04.condainstallcudatoolkit==10.1.243-y
condainstallcudnn==7.6.5-yGetting StartedStep 1.Generate training and testing data. Note: remove--dispfor headless mode.pythonravens/demos.py--assets_root=./ravens/environments/assets/--disp=True--task=block-insertion--mode=train--n=10pythonravens/demos.py--assets_root=./ravens/environments/assets/--disp=True--task=block-insertion--mode=test--n=100Step 2.Train a model e.g., Transporter Networks model.pythonravens/train.py---assets_root=./ravens/environments/assets/-task=block-insertion--agent=transporter--n_demos=10Step 3.Evaluate a Transporter Networks agent with the trained model.pythonravens/test.py--assets_root=./ravens/environments/assets/--disp=True--task=block-insertion--agent=transporter--n_demos=10--n_steps=1000Step 4.Plot results.pythonravens/plot.py--assets_root=./ravens/environments/assets/-disp=True--task=block-insertion--agent=transporter--n_demos=10Optional.Track training and validation losses with Tensorboard.python-mtensorboard.main--logdir=logs# Open the browser to where it tells you to.Datasets and Pre-Trained ModelsDownload our generated train and test datasets and pre-trained models.wgethttps://storage.googleapis.com/ravens-assets/checkpoints.zip
wgethttps://storage.googleapis.com/ravens-assets/block-insertion.zip
wgethttps://storage.googleapis.com/ravens-assets/place-red-in-green.zip
wgethttps://storage.googleapis.com/ravens-assets/towers-of-hanoi.zip
wgethttps://storage.googleapis.com/ravens-assets/align-box-corner.zip
wgethttps://storage.googleapis.com/ravens-assets/stack-block-pyramid.zip
wgethttps://storage.googleapis.com/ravens-assets/palletizing-boxes.zip
wgethttps://storage.googleapis.com/ravens-assets/assembling-kits.zip
wgethttps://storage.googleapis.com/ravens-assets/packing-boxes.zip
wgethttps://storage.googleapis.com/ravens-assets/manipulating-rope.zip
wgethttps://storage.googleapis.com/ravens-assets/sweeping-piles.zipThe MDP formulation for each task uses transitions with the following structure:Observations:raw RGB-D images and camera parameters (pose and intrinsics).Actions:a primitive function (to be called by the robot) and parameters.Rewards:total sum of rewards for a successful episode should be =1.Info:6D poses, sizes, and colors of objects.