Package: atlinter
Atlas InterpolationThe Allen Brain Institute hosts a rich database of mouse brain imagery. It
contains a large number of gene expression datasets obtained
through the in situ hybridization (ISH) staining. While for a given gene
a number of datasets corresponding to different specimen can be found, each of
these datasets only contains sparse section images that do not form a
continuous volume. This package explores techniques that allow to interpolate
the missing slices and thus reconstruct whole gene expression volumes.InstallationPython Version and EnvironmentInstall "Atlas Interpolation"DataRemote Storage AccessModel CheckpointsSection Images and DatasetsNew ISH datasets (advanced, optional)ExamplesPair InterpolationOptical Flow ModelsPredict an Entire Gene Volume (longer runtime)VendorsFunding & AcknowledgmentInstallationPython Version and EnvironmentNote that due to some of our dependencies we're currently limited to python
version3.7. Please make sure you set up a virtual environment with that
version before trying to install this library. If you're unsure how to do that
please have a look atcondaorpyenv.If you are part of the Blue Brain Project and are working on the BB5 you can
find the correct python version in the archive modules betweenarchive/2020-02andarchive/2020-12(inclusive). Here's an example of a set of commands
that will set up your environment on the BB5:modulepurge
moduleloadarchive/2020-12
moduleloadpython
python-mvenvvenv
../venv/bin/activate
python--versionWe also recommend that you make sure thatpipis up-to-date and that the
packageswheelandsetuptoolsare installed:pipinstall--upgradepipwheelsetuptoolsInstall "Atlas Interpolation"In order to access the data and the example scripts a local clone of this
repository is required. Run these commands to get it:gitclonehttps://github.com/BlueBrain/atlas-interpolationcdatlas-interpolationThe "Atlas Interpolation" package can now be installed directly from the clone
we just created:pipinstall'.[data, optical]'DataThe data for this project is managed by theDVC tooland all
related files are located in thedatadirectory. The DVC tool has already been
installed together with the "Atlas Interpolation" package. Every time you need
to run a DVC command (dvc ...) make sure to change to thedatadirectory
first (cd data).Remote Storage AccessWe have already prepared all the data, but it is located on a remote storage
that is only accessible to people within the Blue Brain Project who have
access permissions to projectproj101. If you're unsure you can test your
permissions with the following command:sshbbpv1.bbp.epfl.ch\"ls /gpfs/bbp.cscs.ch/data/project/proj101/dvc_remotes"Possible outcomes:# Access OKatlas_annotation
atlas_interpolation# Access deniedls:cannotopendirectory[...]:PermissiondeniedDepending on whether you have access to the remote storage in the following
sections you will either pull the data from the remote (dvc pull) or download
the input data manually and re-run the data processing pipelines to reproduce
the output data (dvc repro).If you work on the BB5 and have access to the remote storage then run the
following command to short-circuit the remote access (because the remote is
located on the BB5 itself):cddata
dvcremoteadd--localgpfs_proj101\/gpfs/bbp.cscs.ch/data/project/proj101/dvc_remotes/atlas_interpolationcd..Model CheckpointsMuch of the functionality of "Atlas Interpolation" relies on pre-trained deep
learning models. The model checkpoints that need to be loaded are part of the
data.If you have access to the remote storage (see above) you can pull all model
checkpoints from the remote:cddata
dvcpullcheckpoints/rife.dvc
dvcpullcheckpoints/cain.dvc
dvcpullcheckpoints/maskflownet.params.dvc
dvcpullcheckpoints/RAFT.dvccd..If you don't have access to the remote you need to download the checkpoint files
by hand and put the downloaded data into thedata/checkpointsfolder. You
may not need all the checkpoints depending on the examples you want to run. Here
are the instructions for the four models we use: RIFE, CAIN, MaskFlowNet, and
RAFT:RIFE: download the checkpoint from a shared Google Drive folder by followingthis link.
Unzip the contents of the downloaded file intodata/checkpoints/rife.[ref]CAIN: download the checkpoint from a shared Dropbox folder by followingthis link.
Move the downloaded file todata/checkpoints/cain.[ref]MaskFlowNet: download the checkpoint directly from GitHub by followingthis link.
Rename the file tomaskflownet.paramsand move it todata/checkpoints.[ref]RAFT: download the checkpoint files from a shared Dropbox folder by followingthis link.
Move all downloaded.pthfiles to thedata/checkpoints/RAFT/modelsfolder.[ref]If you downloaded all checkpoints or pulled them from the remote you should
have the following files:data
└── checkpoints
    ├── RAFT
    │   ├── models
    │   │   ├── raft-chairs.pth
    │   │   ├── raft-kitti.pth
    │   │   ├── raft-sintel.pth
    │   │   ├── raft-small.pth
    │   │   └── raft-things.pth
    ├── cain
    │   └── pretrained_cain.pth
    ├── maskflownet.params
    └── rife
        ├── contextnet.pkl
        ├── flownet.pkl
        └── unet.pklSection Images and DatasetsThe purpose of the "Atlas Interpolation" package is to interpolate missing
section images within section image datasets. This section explains how to
obtain these data.Remember that if you don't have access to the remote storage (see above) you'll
need to use thedvc reprocommands that download/process the data live. If
you do have access, you'll usedvc pullinstead, which is faster.Normally it's not necessary to get all data. Due to its size it may take a lot
of disk space as well as time to download and pre-process. If you still decide
to do so you can by runningdvc reproordvc pullwithout any parameters.Specific examples only require specific data. You can use DVC to list all data
pipeline stages to find out which stage produces the data you're interested in.
To list all data pipeline stages run:cddata
dvcstagelistIf, for example, you need data located indata/aligned/coronal/Gad1, then
according to the output of command above the relevant stage is namedalign@Gad1. Therefore, you only need to run this stage to get the necessary
data (replacereprobypullif you can access the remote storage):dvcreproalign@Gad1New ISH datasets (advanced, optional)If you're familiar with the AIBS data that we're using and would like to add
new ISH gene expressions that are not yet available as one of our pipeline
stages (check the output ofdvc stage list) then follow the following
instructions.Edit the filedata/dvc.yamland add the new gene name to the lists in thestages:download_dataset:foreachandstages:align:foreachsections.Run the data downloading and processing pipelines (replaceNEW_GENEby the
real gene name that you used indata/dvc.yaml):dvcreprodownload_dataset@NEW_GENE
dvcreproalign@NEW_GENEExamplesIn this section we showcase several typical use-cases of "Atlas Interpolation":Use pair interpolation to predict an intermediate image between two given
imagesPredict optical flow between any pair of images and use it to morph a third
imageIn a gene expression volume predict missing slices and reconstruct the whole
volumeNote that all models accept both RGB images (shape=(height, width, 3))
and grayscale images (shape=(height, width)).Pair InterpolationThe only data you need for this example is the RIFE model checkpoint. Follow
the instructions in the corresponding section above to get it. If you have
access to the remote data storage it's enough to run the following commands:cddata
dvcpullcheckpoints/rife.dvccd..In this example we start with a pair of imagesimg1andimg2(randomly
generated for example's sake). First use the RIFE model to interpolate between
them in a manual way and find the image in-between (img_middle). Then we
demonstrate the use of thePairInterpolateclass that streamlines the
interpolation procedure. Starting with the same pair of images we iterate the
interpolation three times to produce a stack of seven interpolated images
(interpolated_imgs).importnumpyasnpfromatlinter.vendor.rife.RIFE_HDimportModelasRifeModelfromatlinter.vendor.rife.RIFE_HDimportdeviceasrife_devicefromatlinter.pair_interpolationimportPairInterpolate,RIFEPairInterpolationModel# Get the input imagesimg1=np.random.rand(100,200,3)# replace by real section imageimg2=np.random.rand(100,200,3)# replace by real section image# Get the RIFE interpolation modelcheckpoint_path="data/checkpoints/rife/"# Please change, if neededrife_model=RifeModel()rife_model.load_model(checkpoint_path,-1)rife_model.eval()interpolation_model=RIFEPairInterpolationModel(rife_model,rife_device)# Manually predict middle image between img1 and img2preimg1,preimg2=interpolation_model.before_interpolation(img1=img1,img2=img2)img_middle=interpolation_model.interpolate(img1=preimg1,img2=preimg2)img_middle=interpolation_model.after_interpolation(img_middle)print(img_middle.shape)# Streamline the interpolation using PairInterpolate and predict a stack# of 7 intermediate imagesinterpolator=PairInterpolate(n_repeat=3)interpolated_imgs=interpolator(img1,img2,interpolation_model)print(interpolated_imgs.shape)Optical Flow ModelsThe only data you need for this example is the MaskFlowNet model checkpoint.
Follow the instructions in the corresponding section above to get it. If you
have access to the remote data storage it's enough to run the following
commands:cddata
dvcpullcheckpoints/maskflownet.params.dvccd..This example demonstrates how an optical flow model can be used to compute the
optical flow between a pair of images. It can then be used to warp a third
image. The images in this example are randomly generated. In a realistic setting
they should be replaced by real images.importnumpyasnpfromatlinter.optical_flowimportMaskFlowNet# Instantiate an optical flow model (in this case: MaskFlowNet)checkpoint_path="data/checkpoints/maskflownet.params"net=MaskFlowNet(checkpoint_path)# Prepare random images. Should be replaced by real section imagesimg1=np.random.rand(100,200,3)img2=np.random.rand(100,200,3)img3=np.random.rand(100,200,3)# Predict the optical flow between img1 and img2img1,img2=net.preprocess_images(img1=img1,img2=img2)predicted_flow=net.predict_flow(img1=img1,img2=img2)# Warp a third image using the optical flowpredicted_img=net.warp_image(predicted_flow,img3)print(predicted_img.shape)Predict an Entire Gene Volume (Longer Runtime)The data you need for this example are the RIFE model checkpoint and the Vip
gene expression dataset. To get the RIFE checkpoint follow the instruction in
the corresponding section above. If you have access to the remote data storage
it's enough to run the following commands:cddata
dvcpullcheckpoints/rife.dvccd..As described in the data section above, there are two ways of getting the Vip
gene expression dataset. If you have access to the remote data storage you can
pull it from there:cddata
dvcpulldownload_dataset@Vipcd..If you don't have access then you can re-download it. This should always work,
but may take several minutes:cddata
dvcreprodownload_dataset@Vipcd..In this example with start with a gene expression volume that has missing
section images. First we load the image data and the metadata from disk and
wrap it into aGeneDatasetclass. Then we instantiate the RIFE deep learning
model that will be used for interpolation. We use this model to first predict a
single slice in the volume, then we reconstruct the whole volume by predicting
all intermediate slices. Note that this last step is computation-heavy and might
therefore take some time.importjsonimportnumpyasnpfromatlinter.dataimportGeneDatasetfromatlinter.pair_interpolationimportGeneInterpolate,RIFEPairInterpolationModelfromatlinter.vendor.rife.RIFE_HDimportModelasRifeModelfromatlinter.vendor.rife.RIFE_HDimportdeviceasrife_device# Load the gene expression dataset from diskdata_path="data/sagittal/Vip/1102.npy"# Change the path if neededdata_json="data/sagittal/Vip/1102.json"# Change the path if neededsection_images=np.load(data_path)withopen(data_json)asfh:metadata=json.load(fh)section_numbers=[int(s)forsinmetadata["section_numbers"]]axis=metadata["axis"]# Wrap the data into a GeneDataset classgene_dataset=GeneDataset(section_images,section_numbers,volume_shape=(528,320,456,3),axis=axis,)# Load the RIFE deep learning model that will be used for interpolationcheckpoint_path="data/checkpoints/rife"rife_model=RifeModel()rife_model.load_model(checkpoint_path,-1)rife_model.eval()rife_interpolation_model=RIFEPairInterpolationModel(rife_model,rife_device)# Create a gene interpolatorgene_interpolate=GeneInterpolate(gene_dataset,rife_interpolation_model)# Predict a single section imagepredicted_slice=gene_interpolate.predict_slice(10)print(predicted_slice.shape)# Reconstruct the whole volume. This might take some time.predicted_volume=gene_interpolate.predict_volume()print(predicted_volume.shape)VendorsSome dependencies are not available as packages and therefore had to be
vendored. The vendoring is done using thepy-vendorutility. It's installed
automatically together with thedevextras. You can also install it by hand
viapip install py-vendor==0.1.2.The vendoring is then done using the following command (add--forceto
overwrite existing folders):py-vendorrun--configpy-vendor.yamlSee thepy-vendor.yamlfile for details on the vendor sources and files.Funding & AcknowledgmentThe development of this software was supported by funding to the Blue Brain Project,
a research center of the École polytechnique fédérale de Lausanne (EPFL),
from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.Copyright (c) 2021 Blue Brain Project/EPFL