Saravanan666/pypi-org huggingface dataset:
Package: intake-esgf
intake-esgfBadgesOverviewintake-esgfis anintake-esminspiredpackage under development in ESGF2. The main difference is that in place of querying a static index which is completely loaded at runtime,intake-esgfcatalogs initialize empty and are populated by searching, querying ESGF index nodes.InstallationYou may installintake-esgfusingpip:python-mpipinstallintake-esgfFeaturesFor a full listing of features with code examples, please consult thedocumentation. In brief,intake-esgfaims to hide some of the complexity of obtaining ESGF data and get the user the data as fast as we can.Indices are queried in parallel and report when they fail to return a response. The results are aggregated and presented to the user as apandasDataFrame.The locations of the data are hidden from the user. Internally we track which locations provide the user the fastest transfers and automatically favor them for you.Files are downloaded in parallel into a local cache which mirrors the remote storage directory structure. They are returned to the user as a dictionary ofxarrayDatasets. Your search script then becomes the way you download data as well as how you load it into memory for your analysis.Prior to downloading data, we first check that it is not already available locally. This could be because you had previously downloaded it, but also because you are working on a server that has direct access.Cell measure information is harvested from your search results and automatically included in the returned datasets.
Package: intake-esm
Intake-esmIntake-esmBadgesMotivationOverviewInstallationBadgesCIDocsPackageLicenseCitationMotivationComputer simulations of the Earth’s climate and weather generate huge amounts of data.
These data are often persisted on HPC systems or in the cloud across multiple data
assets of a variety of formats (netCDF,zarr, etc...). Finding, investigating,
loading these data assets into compute-ready data containers costs time and effort.
The data user needs to know what data sets are available, the attributes describing
each data set, before loading a specific data set and analyzing it.Finding, investigating, loading these assets into data array containers
such as xarray can be a daunting task due to the large number of files
a user may be interested in. Intake-esm aims to address these issues by
providing necessary functionality for searching, discovering, data access/loading.Overviewintake-esmis a data cataloging utility built on top ofintake,pandas, andxarray, and it's pretty awesome!Opening an ESM catalog definition file: An Earth System Model (ESM) catalog file is a JSON file that conforms
to theESM Collection Specification. When provided a link/path to an esm catalog file,intake-esmestablishes
a link to a database (CSV file) that contains data assets locations and associated metadata
(i.e., which experiment, model, the come from). The catalog JSON file can be stored on a local filesystem
or can be hosted on a remote server.In[1]:importintakeIn[2]:importintake_esmIn[3]:cat_url=intake_esm.tutorial.get_url("google_cmip6")In[4]:cat=intake.open_esm_datastore(cat_url)In[5]:catOut[5]:<GOOGLE-CMIP6catalogwith4dataset(s)from261asset(s>Search and Discovery:intake-esmprovides functionality to execute queries against the catalog:In[5]:cat_subset=cat.search(...:experiment_id=["historical","ssp585"],...:table_id="Oyr",...:variable_id="o2",...:grid_label="gn",...:)In[6]:cat_subsetOut[6]:<GOOGLE-CMIP6catalogwith4dataset(s)from261asset(s)>Access: when the user is satisfied with the results of their query, they can load data assets (netCDF and/or Zarr stores) into xarray datasets:In[7]:dset_dict=cat_subset.to_dataset_dict()-->Thekeysinthereturneddictionaryofdatasetsareconstructedasfollows:'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'|███████████████████████████████████████████████████████████████|100.00%[2/200:18<00:00]Seedocumentationfor more information.InstallationIntake-esm can be installed from PyPI with pip:python-mpipinstallintake-esmIt is also available fromconda-forgefor conda installations:condainstall-cconda-forgeintake-esm
Package: intake-geopandas
intake_geopandasintake_geopandas:Geopandasplugin forIntakeSeeIntake docs.Inintake_geopandas, there are plugins provided for reading geospatial datasets into a geopandas dataframe.
It currently supports reading from the following data sources:GeoJSON filesparquet filesPostGIS databasesESRI ShapefilesSpatialite databasesInstallationThe pip install instructions are:pip install intake_geopandas
Package: intake-hive
Welcome to the Intake Hive pluginThisIntakeplugin
:Example where the Hive table is user_events_hive partitioned by userid:sources:user_events_hive:driver:hiveargs:urlpath:'user_events_yaml_catalog?userid={{userid}}'importpandasaspdimportintakecatalog=intake.open_catalog(catalog_path)# Reads partition userid=42pandas_df:pd.DataFrame=catalog.entity.user.user_events_partitioned(userid="42").read()
Package: intake-informaticslab
Intake Informatics LabIntake catalogues and associated drivers which provide access to hundreds of terabytes of Met Office data.LicencePlease note that individual datasets have different licences and not all are fully open, e.g. some prohibit non-commercial use.Please ensure that you fully understand the licence terms of the dataset before using the data.If using the catalogues made available in this package, the licences are describedin the 'Datasets' section below.
If accessing the data directly, or creating your own catalogue (and/or instance of the driver), the licence is found inLICENCE.txtorLICENCE.mdat the data root.The code in this repository is made available under the MIT Licence.Datasetsair_qualityGridded dataset of air quality approximations for the UK from 2020-01-01 to present. Created from multiple forecasts, and stitched into a continuous time-series. For more information seeair quality technical details.LicenceUsers are required to acknowledge the Met Office as the source of these data by including the following attribution statement in any resulting products, publications or applications:
"Contains Met Office data licensed under the Open Government Licence v3.0"These data are made available under anOpen Government Licence.weather_continuous_timeseriesUK numerical weather prediction output from the UK Met Office. Data are from the very early time steps of the model following data assimilation, stitched into a continuous time series from many forecast model runs. These data were designed to approximate an observation dataset. Covers the UK from 2020-01-01 to present.See the dataREADMEand thetechnical docs.Note:The underlying dataset contains global as well as UK data, but the catalogue provided in this package currently only offers the UK data.LicenceUsers are required to acknowledge the Met Office as the source of these data by including the following attribution statement in any resulting products, publications or applications:
"Contains Met Office data licensed under the Open Government Licence v3.0"These data are made available under anOpen Government Licence.weather_forecastsWeather forecast model output from theMet Office's MOGREPS-UK and MOGREPS-G models, delayed by 24 hours.LicenceThese data are made available under a Creative Commons licence (https://creativecommons.org/licenses/by-nc-nd/4.0/).The data are free to use for non-commercial purposes under the terms of the licence.
You must ensure that you acknowledge the source of this data as prescribed by this licence.
Data or any derivatives made using the data are for internal use only and may not be distributed or otherwise shared.ExamplesSee thebinderfolder for example Notebooks.These example can run as aMyBinder instancebut will run better on a dedicated instance close to the data (Azure East US 2 region).InstallingPyPIpipinstallintake_informaticslabAnacondacondainstall-cconda-forge-cinformaticslabintake_informaticslab
Package: intakeinformer
intakeinformerA nutritional tracker warpped around the USDA FoodData Central API. The warpper allows users to track their daily food intake and analyze the nutritional content of their diet based on the USDA food database and their own selections.Getting Started with IntakeInformerStep 1: Obtain a USDA FoodData Central API KeyRegister for an API Key:Visit the USDA FoodData Central API'sregistration page.Fill in the required details and submit the form to register.Once registered, you will receive an API key. Keep this key secure as it is your personal access token to the API.Step 2: Set the API Key as an Environment VariableSetting the API key as an environment variable keeps it secure and easily accessible by the package.For Unix/Linux/macOS:Open your terminal.Run the following command, replacing'your_api_key_here'with the actual API key you received:exportPOETRY_FDC_API_KEY='your_api_key_here'To ensure the variable is set, you can runecho$POETRY_FDC_API_KEYIt should display your API key.For Windows:Open Command Prompt:You can search for "cmd" or "Command Prompt" in the Windows search bar and open it.Set the Environment Variable:Run the following command in Command Prompt:set POETRY_FDC_API_KEY=your_api_key_hereReplaceyour_api_key_herewith your actual API key.Verify the Environment Variable:To check if the environment variable is set correctly, run:echo %POETRY_FDC_API_KEY%This should display your API key.Step 3: Install and Use IntakeInformerInstall the Package:Use pip (or poetry if you're using it) to installIntakeInformer.Command:pip install IntakeInformer(or the appropriate command if using poetry).Using IntakeInformer:Import and use the functions from IntakeInformer in your Python scripts or interactive environment.Example:fromintakeinformerimportget_calories_for_food_query,dri_benchmark# Use the functions as neededStep 4: TroubleshootingAPI Key Issues:If you encounter issues related to the API key (such as it not being recognized), ensure that you have correctly set the environment variable and that your current session has access to it.Environment Variables:Remember that environment variables set in a terminal session are usually temporary. If you restart your system, you might need to set the environment variable again, unless you add it to your system's permanent profile.Step 5: Seeking HelpIf you run into any issues or have questions, refer to the documentation ofIntakeInformeror reach out for support as provided by the package.UsageLink to the example.ipynb fileContributingInterested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.Licenseintakeinformerwas created by Junchen Xiong. It is licensed under the terms of the MIT license.Creditsintakeinformerwas created withcookiecutterand thepy-pkgs-cookiecuttertemplate.
Package: intake-nested-yaml-catalog
Welcome to Intake plugin for nested YAML catalogsThis is anIntakeplugin supporting a
single YAML hierarchical catalog to organize datasets and avoid a data swamp.Example of organizing the datasets by business domain entities:metadata:hierarchical_catalog:trueentity:customer:customer_attributes:args:urlpath:s3://foodriver:parquetuser:user_profile:args:urlpath:s3://foodriver:parquetCan be accessed as:df=catalog.entity.customer.customer_attributes.read()
Package: intake-odbc
Intake-ODBCODBC Plugin for IntakeOpen Database Connectivity (ODBC) provides a standardised interface to
many data-base systems and other data sources capable of similar
functionality. It passes an SQL text query to the backend via a driver
and routes the resultant data back to the calling program.ODBC can access data in several Data Source Names (DSNs), each of which
constitutes a specific connection to a specific database system of a
given type.User InstallationTo install the intake-odbc plugin, execute the following commandconda install -c intake intake-odbcIn most cases, the end-users of ODBC data do not configure their
own DSNs, merely reference an existing system-wide configuration.
Each DSN will be referenced by a unique name, and in general
have different requirements in terms of further required parameters.
All of this may be defined in the Intake catalog, so that each
catalog entry executes a query on some ODBC backend without the user
having to know how this is achieved.In order to set up the ODBC system on a machine, general guidance can
be found on theturbodbcpages (this is the library used by this
plugin):odbc confguration,database configuration.Developer InstallationCreate a development environment withconda create. Then install the dependencies:conda install -c intake intake turbodbc pandas unixodbcDevelopment installation:git clone https://github.com/ContinuumIO/intake-odbc
cd intake-odbc
python setup.py develop --no-depsCreate a DB to connect to, if you do not have an existing ODBC
configuration. The moduletests/util.pycontains functions to
start and stop MS SQL and PostgreSQL docker images that might be
useful. You will generally need a driver, appropriate for your
system and the DB backend you wish to connect to.Set up odbc config, e.g., the two.inifiles provided. The
default location of these files is system wide at/usr/local/etc,
but you can override this by specifying an environment variable
before starting python, e.g.,export ODBCSYSINI=/path/to/intake-odbc/examples
Package: intake-omnisci
intake-omnisciAnOmniScidriver forintake.InstallationYou can install this driver by runningpip install intake-omnisciIntake should automatically be able to pick it up and use it when encountering
an OmniSci data source in a catalog.UsageSee an example of how to use the driver inthisnotebook.Development EnvironmentCreate development Python environment.Install conda:conda install conda=4.5Install jupyter:conda install jupyterInstallintake:conda install -c intake intakepip install -r requirements.txtpython setup.py developVerify development environment by running the unit tests withpy.test.
Package: intake-parquet
# Intake-parquet[![Build Status](https://travis-ci.org/ContinuumIO/intake-parquet.svg?branch=master)](https://travis-ci.org/ContinuumIO/intake-parquet)
[![Documentation Status](https://readthedocs.org/projects/intake-parquet/badge/?version=latest)](http://intake-parquet.readthedocs.io/en/latest/?badge=latest)[Intake data loader](https://github.com/ContinuumIO/intake/) interface to the parquet binary tabular data format.Parquet is very popular in the big-data ecosystem, because it provides columnar
and chunk-wise access to the data, with efficient encodings and compression. This makes
the format particularly effective for streaming through large subsections of even
larger data-sets, hence it’s common use with Hadoop and Spark.Parquet data may be single files, directories of files, or nested directories, where
the directory names are meaningful in the partitioning of the data.### FeaturesThe parquet plugin allows for:efficient metadata parsing, so you know the data types and number of records without
loading any datarandom access of partitionscolumn and index selection, load only the data you needpassing of value-based filters, that you only load those partitions containing some
valid data (NB: does not filter the values within a partition)### InstallationThe conda install instructions are:` conda install-cconda-forgeintake-parquet`### ExamplesSee the notebook in the examples/ directory.
Package: intake-pattern-catalog
Intake Pattern Catalogintake-pattern-catalog is apluginfor Intake
which allows you to specify a file-path pattern which can represent a number of different entries.Note that this is different from the patterns you can write with the csv driver which get turned into a single entryInstallation instructionspipinstallintake-pattern-catalog# orcondainstallintake-pattern-catalogUsageUsedriver: pattern_catto use this driver in your catalogs.Consider the following list of files in an S3 bucket:bucket-name/folder/a_1.csvbucket-name/folder/b_1.csvbucket-name/folder/c_1.csvbucket-name/folder/a_2.csvbucket-name/folder/b_2.csvAnd the following catalog definition yaml file:---metadata:version:1sources:stuff:description:Stuff and thingsdriver:pattern_catargs:urlpath:"s3://bucket-name/folder/{foo}_{bar}.csv"driver:csvDerived datasetsIf you would like to create aderived datasetbased on apattern_catdataset, you can usedriver: pattern_cat_transform, which will apply
a transformation function to each entry returned byget_entry. For example, you can
add to the above example yaml file:stuff_transformed:description:Everything in stuff, doubleddriver:pattern_cat_transformargs:targets:-stufftransform:"path.to.doubling_function"Catalog APIAccess entry by kwargs:>catalog.stuff.get_entry(foo='a',bar=1)sources:foo_a_bar_1:args:storage_options:use_listings_cache:falseurlpath:s3://bucket-name/folder/a_1.csvdescription:''driver:intake.source.csv.CSVSourcemetadata:catalog_dir:...Note that this could also be accessed withcatalog.stuff.foo_a_bar_1See all valid kwarg combinations:>catalog.stuff.get_entry_kwarg_sets()[{"foo":"a","bar":"1"},{"foo":"b","bar":"1"},{"foo":"c","bar":"1"},{"foo":"a","bar":"2"},{"foo":"b","bar":"2"},]CachingThe default way of controlling any caching with a pattern-catalog is using attl(in seconds),
which is an optional value underargswhich specifies how long should wait after fetching a list of files
which match the pattern before it loads them again. The defaultttlis 60 seconds.
If you want to force it to always get the latest list of available entries, set thettlto 0.
Package: intake-sdmx
2021 Dr. LeoUseful links:Source code @ Github—intake_sdmx documentationintake_sdmxis anApache 2.0-licensedintakeplugin for SDMX-compliant data sources
such as BIS, ECB, ESTAT, INSEE, ILO, UN, UNICEF, World Bank and more.intake_sdmxleveragespandaSDMX.LicenseCopyright 2021, Dr. LeoLicensed under the Apache License, Version 2.0 (the “License”); you may not use
these files except in compliance with the License. You may obtain a copy of the
License:from the file LICENSE included with the source code, orathttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
Package: intake-stac
Intake-STACThis is anIntakedata source forSpatioTemporal Asset Catalogs (STAC). The STAC specification provides a common metadata specification, API, and catalog format to describe geospatial assets, so they can more easily indexed and discovered. A 'spatiotemporal asset' is any file that represents information about the earth captured in a certain space and time.Intake-STAC provides an opinionated way for users to load Assets from STAC catalogs into the scientific Python ecosystem. It uses theintake-xarrayplugin and supports several file formats including GeoTIFF, netCDF, GRIB, and OpenDAP.InstallationIntake-STAC has a fewrequirements, such asIntake,intake-xarrayandpystac. Intake-stac can be installed in any of the following ways:We recommend installing the latest release withconda:$condainstall-cconda-forgeintake-stacOr the latest development version withpip:$pipinstallgit+https://github.com/intake/intake-stacQuickstartimportintakecatalog_url='https://www.planet.com/data/stac/catalog.json'cat=intake.open_stac_catalog(catalog_url)collection=cat['planet-disaster-data']subset=collection['hurricane-harvey']['hurricane-harvey-0831']item=subset['Houston-East-20170831-103f-100d-0f4f-RGB']da=item['thumbnail'].to_dask()daTheexamples/directory contains several Jupyter Notebooks illustrating common workflows.STAC Indexis a convenient website for finding datasets with STACsVersionsTo install a specific version of intake-stac, specify the version in the install commandpipinstallintake-stac==0.4.0The table below shows the corresponding versions between intake-stac and STAC:intake-stacSTAC0.2.x0.6.x0.3.x1.0.0-betaX0.4.x1.0.0Aboutintake-stacwas created as part of thePangeoinitiative under support from the NASA-ACCESS program. See the initialdesign document.
Package: intake-thredds
intake-threddsCIDocsPackageLicenseIntake interface toTHREDDSdata catalogs.InstallationIntake-thredds can be installed from PyPI with pip:python-mpipinstallintake-threddsIntake-thredds is also available from conda-forge for conda installations:condainstall-cconda-forgeintake-threddsSeedocumentationfor more information.
Package: intake-xarray
intake-xarrayIntake-xarray: xarray Plugin forIntakeSeeIntake docsfor a general introduction and usage
of Intake and theintake-xarray docsfor details specific to the
data drivers included in this package.Inintake-xarray, there are plugins provided for reading data intoxarraycontainers:NetCDF (also handles other file formats which can be passed toxarray.open_datasetsuch as grib)OPeNDAPRasterioZarrimagesand it provides the ability to read xarray data from an Intake server.InstallationThe conda install instructions are:conda install -c conda-forge intake-xarrayTo install optional dependencies:conda install -c conda-forge pydap rasterio
Package: intbitset
Installationintbitset is on PyPIpip install intbitsetWe provide pre-built wheels for the most common operating systems and common 64 bits CPU
architectures. Otherwise, you will need a C compiler if you build from sources.DocumentationTheintbitsetlibrary provides a set implementation to store sorted
unsigned integers either 32-bits integers (between0and2**31- 1orintbitset.__maxelem__) or an infinite range
with fast set operations implemented via bit vectors in aPython C
extensionfor speed and reduced memory usage.Theinbitsetclass emulates the Python built-in set class interface
with some additional specific methods such as its own fast dump and load
marshalling functions.>>> from intbitset import intbitset
>>> x = intbitset([1,2,3])
>>> y = intbitset([3,4,5])
>>> x & y
intbitset([3])
>>> x | y
intbitset([1, 2, 3, 4, 5])Additionally,intbitsetsupports:Thepickle protocol,Theiterator protocolintbitsetcan behave like asequencethat can be sliced.Natural min and max. Because the integers are always stored sorted, the fist
element of a non-empty set[0]is also themin()integer and the last
element[-1]is also themax()integer in the set.When compared to the standard librarysetclass,intbitsetset
operations and the intersection, union and difference ofintbitset``canbe up to 5000 faster for dense integer sets than the standard library ``set.Complete documentation is available at <http://intbitset.readthedocs.io> or
can be built using Sphinx:pip install sphinx
python setup.py build_sphinxTestingRunning the tests are as simple as:pip install -e .[tests]
pytestRunning the tests on multiple Python versions:pip install tox
toxDevelopmentTo regenerate the C code with Cython:pip install cython
cython intbitset/intbitset.pyxThen commit the regenrated C source and update the CHANGELOG.rstLicenseCopyright (C) CERN and othersSPDX-License-Identifier: LGPL-3.0-or-laterintbitset is free software; you can redistribute it and/or modify it under the
terms of the GNU Lesser General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.intbitset is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more details.You should have received a copy of the GNU Lesser General Public License along with
intbitset; if not, write to the Free Software Foundation, Inc., 59 Temple
Place, Suite 330, Boston, MA 02111-1307, USA.In applying this licence, CERN does not waive the privileges and immunities
granted to it by virtue of its status as an Intergovernmental Organization or
submit itself to any jurisdiction.