Package: pylops-distributed
:vertical_traffic_light: :vertical_traffic_light: This library is under early development.
Expect things to constantly change until version v1.0.0. :vertical_traffic_light: :vertical_traffic_light:ObjectiveThis library is an extension ofPyLopsfor distributed operators.As much asnumpyandscipylie
at the core of the parent project PyLops, PyLops-distributed heavily builds on top ofDask, a Python library for distributed computing.Doing so, linear operators can be distributed across several processes on a single node
or even across multiple nodes. Their forward and adjoint
are first lazily built as directed acyclic graphs and evaluated only when requested by
the user (or automatically within one of our solvers).Here is a simple example showing how a diagonal operator can be created,
applied and inverted using PyLops:importnumpyasnpfrompylopsimportDiagonaln=10x=np.ones(n)d=np.arange(n)+1Dop=Diagonal(d)# y = Dxy=Dop*x# x = D'yxadj=Dop.H*y# xinv = D^-1 yxinv=Dop/yand similarly using PyLops-distributed:importnumpyasnpimportdask.arrayasdaimportpylops_distributedfrompylops_distributedimportDiagonal# set-up clientclient=pylops_distributed.utils.backend.dask()n=10x=da.ones(n,chunks=(n//2,))d=da.from_array(np.arange(n)+1,chunks=(n//2,n//2))Dop=Diagonal(d)# y = Dxy=Dop*x# x = D'yxadj=Dop.H*y# xinv = D^-1 yxinv=Dop/yda.compute((y,xadj,xinv))client.close()It is worth noticing two things at this point:in this specific case we did not even need to reimplement theDerivativeoperator.
Calling numpy operations as methods (e.g.,x.sum()) instead of functions (e.g.,np.sum(x))
makes it automatic for our operator to act as a distributed operator when a dask array is provided instead. Unfortunately not all numpy functions are also implemented as methods: in those cases we
reimplement the operator directly within PyLops-distributed.Using*and.H*is still possible also within PyLops-distributed, however when initializing an
operator we will need to decide whether we want to simply create dask graph or also evaluation.
This gives flexibility as we can decide if and when apply evaluation using thecomputemethod
on a dask array of choice.Getting startedYou needPython 3.5 or greater.From PyPiComing soon...From GithubYou can also directly install from the master nodepip install https://git@github.com/equinor/pylops-distributed.git@masterContributingFeel like contributing to the project? Adding new operators or tutorial?Follow the instructions fromPyLops official documentation.DocumentationThe official documentation of PyLops-distributed is availablehere.Moreover, if you have installed PyLops using thedeveloper environmentyou can also build the documentation locally by
typing the following command:make docOnce the documentation is created, you can make any change to the source code and rebuild the documentation by
simply typingmake docupdateNote that if a new example or tutorial is created (and if any change is made to a previously available example or tutorial)
you are required to rebuild the entire documentation before your changes will be visible.HistoryPyLops-Distributed was initially written and it is currently maintained byEquinor.
It is an extension ofPyLopsfor large-scale optimization withdistributedlinear operators that can be tailored to our needs, and as contribution to the free software community.ContributorsMatteo Ravasi, mrava87