Package: pandata
Pandata: The scalable open-source analysis stackPandata is a fully open source, high-performance, modern Python data-analytics stack usable in any scientific, engineering, or analytical domain.Are you limited by your domain’s software stack?Every scientific or engineering discipline has its own computing needs.
Many such disciplines have developed entirely separate sets of tools for:Storing dataReading dataProcessing dataPlotting dataAnalyzing dataModeling dataExploring dataThese stacks are largely tied to outdated architectures and assumptions:Not cloud or remote friendly: Tied to a local desktop GUI or OSNot scalable: Tied to a single processor (CPU), whether for technical or licensing reasonsNot general purpose: Narrow audience, maintained by few, and unknown outside the fieldIt’s all just data – time for a better way!Pandata: the scalable open-source analysis stackInstead of your outdated stack, use modern Python data-science tools that are:Domain independent:Maintained, used, and tested by many people all across the worldEfficient:Run at machine-code speeds using vectorized data or JIT compilationScalable:Run on anything from a single-core laptop to a thousand-node clusterCloud friendly:Fully usable for local or remote compute using data on any file storage systemMulti-architecture:Runs on your desktop and on Mac/Windows/Linux CPUs and GPUsScriptable:Run in batch mode for parameter searches and unattended operationCompositional:Select which tools you need and put them together to solve your problemVisualizable:Support rendering even the largest datasets without conversion or approximationInteractive:Support fully interactive exploration, not just rendering static images or text filesShareable:Deployable as web apps for use by anyone anywhereOSS:Free, open, and ready for research or commercial use, without restrictive licensingBut I don’t do data science, you say? You do! Data science is what’s shared across lots of disciplines; it’s not just for AI and ML (though it supports those well too!)What is Pandata and why do I need it?Pandata is just a name for a specific collection of Python libraries maintained separately by different people. Pandata libraries are designed to work well with each other to achieve the goals listed above (being scalable, interactive, etc.). You don't need Pandata for anything other than to know which libraries are designed to work well together in this way. Just use any library from Pandata and be happy, knowing that if you need something covered by one of the other libraries, you can use them together without jeopardizing scalability, interactivity, and so on.Who runs Pandata?Pandata is just this informational website set up by the authors of some of the Pandata tools; there's no management or policies or software development specifically associated with Pandata. But if you have questions or ideas about what to do with Pandata, feel free to open an issue for discussion!ExamplesThere are lots of examples online of applying Pandata libraries to solve problems, including:Pangeo: JupyterHub, Dask, Xarray for climate science; Pandata is Pangeo but for any field!Project Pythia: Geoscience-specific examples, many of which use Pandata toolsAttractors: Processing huge datasets with Numba, rendering with DatashaderCensus: Reading chunked data from Parquet, rendering with Dask + DatashaderShip traffic: Rendering spatially indexed data with interactive lookupLandsat: Intake data catalog, xarray n-D data, hvPlot+Bokeh plotting, Dask MLMinian: Jupyter, Dask, and HoloViews for neuroscienceSee thePandata paper from SciPy 2023for all the details, and then just download and use any of the packages in Pandata in any combination and enjoy having all this power at your fingertips!