Package: deepmreye
DeepMReye: magnetic resonance-based eye tracking using deep neural networksThisJupyter Notebookprovides a step-by-step walkthrough of the code. It includes eyeball coregistration, voxel extraction, model training and test as well as basic performance measures. Alternatively, here is aColab Notebook.ThisData Repositoryincludes exemplary data for model training and test, source data of all paper figures as well as pre-trained model weights.Moreover, here are additionalUser Recommendationsas well as aFrequently-Asked-Questions (FAQ)page. If you have other questions, please reach out to us.InstallationOption 1: Pip installPip installationInstall DeepMReye with a CPU/GPU version ofTensorFlowusing the following command.pip install deepmreyeAnaconda / Miniconda installationTo encapsulate DeepMReye in a virtual environment install with the following commands:conda create --name deepmreye python=3.9
conda activate deepmreye
pip install deepmreyeFor GPU support, follow tensorflowinstallinstruction, e.g.:conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0If installation ofANTsPyfails try to manually install it via:git clone https://github.com/ANTsX/ANTsPy
cd ANTsPy
pip install CMake
python3 setup.py installOption 2: ColabWe provide aColab Notebookshowcasing model training and evaluation on a GPU provided by Google Colab. To use your own data, preprocess your data locally and upload only the extracted eyeball voxels. This saves space and avoids data privacy issues. See theJupyter Notebookfor the preprocessing and eyeball-extraction code.Option 3: DockerPull the image from docker hub.dockerpulldeepmreye/deepmreyeUse deepMReye in a docker container via jupyterlab:mkdir-p$PWD/notebooks
dockerrun-it--rm\--publish8888:8888\--volume$PWD/notebooks:/home/neuro/notebooks\deepmreye/deepmreye:latest\jupyter-lab--no-browser--ip0.0.0.0Option 4: Streamlit (Browser version)If you would just like to try it out and decode gaze coordinates in your data using a pretrained model, the easiest way is using our streamlit app.Running the following commands will open a browser window that allows you to upload your data (.nii or .nii.gz) and then download the corresponding gaze coordinates shortly after.
Please read theUser Recommendationsbefore using the pretrained models.gitclonehttps://github.com/DeepMReye/DeepMReye.gitcdDeepMReye
pipinstall.
pipinstallstreamlit
streamlitrunstreamlit/streamlit.pyData formatsThefMRI datashould be organized in 4D NIFTI files (.nii), containing the realigned 3D images acquired over time. The pipeline then extracts the eyeball voxels automatically and saves them as Python Pickle files, which serve as model input. For model training, you additionally needtraining labels, a numpy array containing 10 gaze coordinates per functional volume. These gaze coordinates can either be camera-based eye-tracking labels or the coordinates of a fixation target, and many file formats can be easily read (e.g. .npy, .npz, .mat, .csv etc.).Please see ourFAQpage for more details on data formats and preprocessing.Hardware requirementsThe GPU version of DeepMReye requires a NVIDIA GPU.Software requirementsThe following python dependencies are being automatically installed when installing DeepMReye (specified in setup.cfg):tensorflow-gpu (2.2.0)
numpy (1.19.1)
pandas (1.0.5)
matplotlib (3.2.2)
scipy (1.5.0)
ipython (7.13.0)
plotly (4.14.3)Version in parentheses indicate the ones used for testing the framework. Its extensively tested on Linux 16.04 but should run on all OS (Windows, Mac, Linux) supporting a Python version >3.8 and pip. It is recommended to install the framework and dependencies in a virtual environment (e.g. conda).BIDS appIf you would like to run a pretrained version of DeepMReye on datasets that follow the Brain Imaging Data Structure (BIDS) format, check outbidsMReye: a wrapper for DeepMReye for BIDS datasets (incl.fMRIprepoutputs) developed byRemi Gau. Please read the paper andUser Recommendationsbefore you use it.CorrespondenceIf you have questions, comments or inquiries, please check out the onlineUser documentionand reach out to us: markus.frey[at]ntnu.no & matthias.nau[at]ntnu.no