Package: gwaihir
WelcomeContact :david.simonne@universite-paris-saclay.frYou can install the latest version of the package by cloning the repository and via thesetup.pyscript (pip install .)You can install gwaihir by typingpip install -U gwaihirin your command line. A stable version from the master branch uploaded to pypi.org will be used (https://pypi.org/project/gwaihir/)Otherwise, you can install the latest commit of the package by cloning this repository and typingpip install .in the terminal (you must be in the same directory as thesetup.pyscript), this will allow you to have the latest updates.Here is a link to a poster that tries to present Gwaihir:Poster_Gwaihir.pdfAnd to the paper:Simonne, D., Carnis, J., Atlan, C., Chatelier, C., Favre-Nicolin, V., Dupraz, M., Leake, S.  J., Zatterin, E., Resta, A., Coati, A. & Richard, M.  I. (2022). J. Appl. Cryst. 55, 1045-1054Important code snippetsTo increase the width of the cells in Jupyter Notebook:fromIPython.core.displayimportdisplay,HTMLdisplay(HTML("<style>.container { width:75% !important; }</style>"))To avoid automatic cell scrolling:%%javascriptIPython.OutputArea.prototype._should_scroll=function(lines){returnfalse;}To open the GUI:fromgwaihir.guiimportInterfaceGUI=Interface()GUI Preview:Pre-processing datahttps://user-images.githubusercontent.com/51970962/154160601-f3e7878a-d2c6-4560-95e5-adf7087f59ab.mp4Phase retrievalhttps://user-images.githubusercontent.com/51970962/154160830-f3c6460b-14e5-4bcc-99f5-e8691278a4e9.mp4Data plottinghttps://user-images.githubusercontent.com/51970962/154160549-c5caea1b-afa0-4a29-a5a8-aff8a1a5158b.mp4Post-processinghttps://user-images.githubusercontent.com/51970962/154236802-24643473-1ee9-4d01-823c-beca07ea1c58.mp4An example file can be downloaded at:https://www.dsimonne.eu/Attachments/align_031968.cxiKnown bogThe first time you installgwaihir, it is possible that when you open the Interface, you only see some text printed, or the content of the readme tab, but you cannot interact with anything. This is solved by restarting your computer.Bog with printed text:Bog with README tab:Clusters at ESRFGwaihironlyworks with the p9 partition at the ESRF, optimized for phase retrieval.If you want to use it for data analysis, you can installgwaihirandbcdion rnice.Jupyter-slurmHow to access:Web browser:https://jupyter-slurm.esrf.fr/hub/spawnTerminal (for advanced users) :ssh -X <login>@slurm-nice-develAsk for a GPU:srun -N 1 --partition=p9gpu --gres=gpu:1 --time=06:00:00 --pty bashAvailable environements/usr/bin/python3: your personal environemntp9.dev : optimised for BCDI, gwaihir and PyNX, development version,source /data/id01/inhouse/david/p9.dev/bin/activatep9.stable : optimised for BCDI, gwaihir and PyNX, stable version,source /data/id01/inhouse/david/p9.stable/bin/activatep9.pynx-devel : pynx only, frequently updated :source /sware/exp/pynx/devel.p9/bin/activateYou are not allowed tomodifythese environments but you canlinka kernel if you wish tousethem in jupyter.To do so:Source the environment; e.g.source /data/id01/inhouse/david/p9.dev/bin/activateMake sure that:you are on slurmyou requested a GPUCreate the kernel:python3 -m ipykernel install --user --name p9.stableDocumentationOnce you feel confident, you should create your own environment, to avoid sudden updates that may impact your work!To list the kernels you have installed:jupyter kernelspec listAnd to remove them:jupyter kernelspec uninstall <kernelname>Make sure that you are using the right kernel on your Jupyter Notebook !Set upsshconnection without using password (mandatory for batch jobs)Login into slurm (make sure that you asked for a GPU)Open a terminal (new -> terminal)Enter the following commands (replace<username>with your username, for me it is simonne)cdssh-keygen -t rsa(press enter when prompted, ~ 3 times)ssh <username>@slurm-nice-devel mkdir -p .sshcat .ssh/id_rsa.pub | ssh <username>@slurm-nice-devel 'cat >> .ssh/authorized_keys'You should not need a password anymore when login into slurm, make sure it is the case by typingssh <username>@slurm-nice-develCLuster at SOLEILGRADESTo analyse data recorded at SOLEIL from your personal computer, you can use Jupyter Notebook via GRADES. The documentation is here (accessible on- site via the SOLEIL wifi or with the SOLEIL VPN) :http://confluence.synchrotron-soleil.fr/display/EG/Service%3A+Jupyter+NotebookUse this link to open Jupyter Notebook :http://grades-01.synchrotron-soleil.fr/notebook/Launch a Terminal (command line) from the upper-right 'New' menu. The system is a Debian 10, so you can enter linux commands there.PyNXis already installed on GRADES, a version that is out of my control. So you 'just' have to download thebcdiandgwaihirpackages by typingpip3 install --proxy=http://195.221.0.35:8080 -U gwaihir bcdiIf you encounter an error with gwaihir or bcdi, it is possible that the pip packages are not up to date. Then you should follow the procedure describedbelowand manually download the packages. Just replacepip installbypip3 install --proxy=http://195.221.0.35:8080,  the proxy IP can be 195.221.0.34:8080 or 195.221.0.35:8080 on the ReS (offices and VPN), and 195.221.10.6:8080 or 195.221.10.7:8080 on the REL (beam-lines, RAS).You can also directly use a virtual machine provided by GRADESGo tohttps://re-grades-01.exp.synchrotron-soleil.fr/qemu-web-desktop/Use the SUNset/LDAP id.Click onCreatethen onConnect(after 10 sec).Once the desktop is available, search "gwaihir" in the bottom left menu and execute the program.You are now connected to the ruche with an environment that has all the necessary packages, you just need to open a notebook.SixSA GPU is installed on sixs3, a computer available on the beamline, for phase retrieval.Please respect the following steps:Make sure that you are logged in ascom-sixsActivate the environmentsource_py3.9orsource /home/experiences/sixs/simonne/Documents/py39-env/bin/activate, this environment is protected and you cannot modify it.Launchjupyter notebookGo to the test_data folder and then choose the beamline you want to testFollow the instructions in the notebookCristalA GPU is installed on cristal4, a computer available on the beamline, for phase retrieval.Please respect the following steps:Make sure that you are logged in ascom-cristalActivate the environmentsource_gwaihirorsource /home/experiences/crystal/com-cristal/PackagesGwaihir/py-gwaihir/bin/activate, this environment is protected and you cannot modify it.Launchjupyter notebookGo to the test_data folder and then choose the beamline you want to testFollow the instructions in the notebookInstalling different packages yourselfFirst, I advise you to create a/Packagesdirectory to keep these.Secondly, I advise you to create a virtual environment to help with debogging, and so that once everything works, you don't update a package by mistake. To do so please follow the following steps:Create a virtual environmentmkdir py38-envcd py38-env/python3.8 -m venv .source bin/activate# To activate the environmentMake surewheelandsetuptoolsare installed:pip install wheel setuptools pip --upgradeThen you should create an alias such as:alias source_p9="source /home/user/py38-env/bin/activate"Specific instructions for the p.9 clusterIfvtkdoes not install (on the p9 cluster at the ESRF for example), you can type :pip install --trusted-host www.silx.org --find-links http://www.silx.org/pub/wheelhouse vtk, you may also need to remove the version requirements inbcdi/setup.pyIfPyQt5does not install (also on the p9 cluster at the ESRF), you can install it by activating your environment from the rnice cluster.1) Install PyNXUse the latest versioncd /Packagesmkdir PyNX_installcd PyNX_install/curl -O http://ftp.esrf.fr/pub/scisoft/PyNX/pynx-devel-nightly.tar.bz2# Installation details within install-pynx-venv.shsource_p9pip install pynx-devel-nightly.tar.bz2[cuda,gui,mpi]# Install with extras cuda, mpi, cdicitePyNX: high-performance computing toolkit for coherent X-ray imaging based on operators is out: J. Appl. Cryst. 53 (2020), 1404, also available asarXiv:2008.115112) Install gwaihircd /Packagesgit clone https://github.com/DSimonne/gwaihir.gitcd gwaihirsource_p9pip install .citeSimonne, D., Carnis, J., Atlan, C., Chatelier, C., Favre-Nicolin, V., Dupraz, M., Leake, S.  J., Zatterin, E., Resta, A., Coati, A. & Richard, M.  I. (2022). J. Appl. Cryst. 55, 1045-1054.3) Install bcdiLatest version tested : v0.2.8cd /Packagesgit clone https://github.com/carnisj/bcdi.gitcd bcdisource_p9pip install .citeDOI: 10.5281/zenodo.32576164) Install facet-analyser (Debian 11 only)Send a thank you email to Fred Picca =Dcd /Packagesgit clone https://salsa.debian.org/science-team/facet-analyser.gitcd facet-analysergit checkoutsudo mk-build-deps -iMake sure that you have qt installed, for me I had to installlibqt5opengl5-dev(debian-testing)debuild -bif the package creation fail, try to ignore the test in /debian/rules (line 19)sudo debiThe package is now installed. You can check the locations of its files with the commanddpkg -L facet-analyserYou should see a file named/usr/lib/x86_64-linux-gnu/paraview-5.9/plugins/FacetAnalyser/FacetAnalyser.soNow launch/usr/bin/paraview(if not installed yet, good luck, refer tohttps://www.paraview.org/Wiki/ParaView:Build_And_Install#Installing)In paraview, go to Tools > Manage Plugins > Load NewHere type the path to the plugin that was printed with thedpkg -L facet-analysercommand.Feel free to add it to/usr/bin/pluginso that it is loaded automatically.citeGrothausmann, R. (2015). Facet Analyser : ParaView plugin for automated facet detection and measurement of interplanar angles of tomographic objects. March.To go further ...UsingGwaihironly as a plotting tool in Jupyter NotebookQuick navigation betweenvtkfiles in the GUI (outdated but it can give you some ideas)It is possible to automate the navigation in the GUI !Here I have a pandas DataFrame that contains data about my scans, I use to automate the navigation:importtimeimportnumpyasnpimportmatplotlib.pyplotaspltimportpandasaspdimportglobdf=pd.read_csv("reconstructions/scans_data.csv")GUI.tab_facet.children[3].value=FalseGUI.window.selected_index=0time.sleep(1)GUI._list_widgets_init_dir.children[7].value=Falsetime.sleep(1)scan=3600row=df[df.scan==scan]particle=row.particle.values[0]temp=row.temp_given.values[0]condition=row.condition.values[0]GUI._list_widgets_init_dir.children[2].value=scanGUI._list_widgets_init_dir.children[3].value=f"/data/id01/inhouse/david/SIXS_June_2021/reconstructions/{temp}/{condition}/{particle}/S{scan}/data/"GUI._list_widgets_init_dir.children[4].value=f"/data/id01/inhouse/david/SIXS_June_2021/reconstructions/{temp}/{condition}/{particle}/"time.sleep(1)GUI._list_widgets_init_dir.children[7].value=Truetime.sleep(1)GUI.window.selected_index=9GUI.tab_facet.children[3].value="load_csv"