Package: pinecone-datasets
Pinecone Datasetsinstallpipinstallpinecone-datasetsUsage - LoadingYou can use Pinecone Datasets to load our public datasets or with your own datasets. Datasets library can be used in 2 main ways: ad-hoc loading of datasets from a path or as a catalog loader for datasets.Loading Pinecone Public Datasets (catalog)Pinecone hosts a public datasets catalog, you can load a dataset by name usinglist_datasetsandload_datasetfunctions. This will use the default catalog endpoint (currently GCS) to list and load datasets.frompinecone_datasetsimportlist_datasets,load_datasetlist_datasets()# ["quora_all-MiniLM-L6-bm25", ... ]dataset=load_dataset("quora_all-MiniLM-L6-bm25")dataset.head()# Prints# ┌─────┬───────────────────────────┬─────────────────────────────────────┬───────────────────┬──────┐# │ id  ┆ values                    ┆ sparse_values                       ┆ metadata          ┆ blob │# │     ┆                           ┆                                     ┆                   ┆      │# │ str ┆ list[f32]                 ┆ struct[2]                           ┆ struct[3]         ┆      │# ╞═════╪═══════════════════════════╪═════════════════════════════════════╪═══════════════════╪══════╡# │ 0   ┆ [0.118014, -0.069717, ... ┆ {[470065541, 52922727, ... 22364... ┆ {2017,12,"other"} ┆ .... │# │     ┆ 0.0060...                 ┆                                     ┆                   ┆      │# └─────┴───────────────────────────┴─────────────────────────────────────┴───────────────────┴──────┘Expected dataset structurepinecone datasets can load dataset from every storage where it has access (using the default access: s3, gcs or local permissions)we expect data to be uploaded to the following directory structure:├── my-subdir                     # path to where all datasets
│   ├── my-dataset                # name of dataset
│   │   ├── metadata.json         # dataset metadata (optional, only for listed)
│   │   ├── documents             # datasets documents
│   │   │   ├── file1.parquet      
│   │   │   └── file2.parquet      
│   │   ├── queries               # dataset queries
│   │   │   ├── file1.parquet  
│   │   │   └── file2.parquet   
└── ...The data schema is expected to be as follows:documentsdirectory contains parquet files with the following schema:Mandatory:id: str, values: list[float]Optional:sparse_values: Dict: indices: List[int], values: List[float],metadata: Dict,blob: dictnote: blob is a dict that can contain any data, it is not returned when iterating over the dataset and is inteded to be used for storing additional data that is not part of the dataset schema. however, it is sometime useful to store additional data in the dataset, for example, a document text. In future version this may become a first class citizen in the dataset schema.queriesdirectory contains parquet files with the following schema:Mandatory:vector: list[float], top_k: intOptional:sparse_vector: Dict: indices: List[int], values: List[float],filter: Dictnote: filter is a dict that contain pinecone filters, for more information seeherein addition, a metadata file is expected to be in the dataset directory, for example:s3://my-bucket/my-dataset/metadata.jsonfrompinecone_datasets.catalogimportDatasetMetadatameta=DatasetMetadata(name="test_dataset",created_at="2023-02-17 14:17:01.481785",documents=2,queries=2,source="manual",bucket="LOCAL",task="unittests",dense_model={"name":"bert","dimension":3},sparse_model={"name":"bm25"},)full metadata schema can be found inpinecone_datasets.catalog.DatasetMetadata.schemaLoading your own dataset from catalogTo set you own catalog endpoint, set the environment variableDATASETS_CATALOG_BASEPATHto your bucket. Note that pinecone uses the default authentication method for the storage type (gcsfs for GCS and s3fs for S3).exportDATASETS_CATALOG_BASEPATH="s3://my-bucket/my-subdir"frompinecone_datasetsimportlist_datasets,load_datasetlist_datasets()# ["my-dataset", ... ]dataset=load_dataset("my-dataset")additionally, you can load a dataset from the Dataset classfrompinecone_datasetsimportDatasetdataset=Dataset.from_catalog("my-dataset")Loading your own dataset from pathYou can load your own dataset from a local path or a remote path (GCS or S3). Note that pinecone uses the default authentication method for the storage type (gcsfs for GCS and s3fs for S3).frompinecone_datasetsimportDatasetdataset=Dataset.from_path("s3://my-bucket/my-subdir/my-dataset")This assumes that the path is structured as described in the Expected dataset structure sectionLoading from a pandas dataframePinecone Datasets enables you to load a dataset from a pandas dataframe. This is useful for loading a dataset from a local file and saving it to a remote storage.
The minimal required data is a documents dataset, and the minimal required columns areidandvalues. Theidcolumn is a unique identifier for the document, and thevaluescolumn is a list of floats representing the document vector.importpandasaspddf=pd.read_parquet("my-dataset.parquet")metadata=DatasetMetadata(**metadata_dict)dataset=Dataset.from_pandas(documents=df,queries=None,metadata=metadata)Please check the documentation for more information on the expected dataframe schema. There's also a column mapping variable that can be used to map the dataframe columns to the expected schema.Usage - Accessing dataPinecone Datasets is build on top of pandas. This means that you can use all the pandas API to access the data. In addition, we provide some helper functions to access the data in a more convenient way.Accessing documents and queries dataframesaccessing the documents and queries dataframes is done using thedocumentsandqueriesproperties. These properties are lazy and will only load the data when accessed.document_df:pd.DataFrame=dataset.documentsquery_df:pd.DataFrame=dataset.queriesUsage - IteratingOne of the main use cases for Pinecone Datasets is iterating over a dataset. This is useful for upserting a dataset to an index, or for benchmarking. It is also useful for iterating over large datasets - as of today, datasets are not yet lazy, however we are working on it.# List Iterator, where every list of size N Dicts with ("id", "values", "sparse_values", "metadata")dataset.iter_documents(batch_size=n)# Dict Iterator, where every dict has ("vector", "sparse_vector", "filter", "top_k")dataset.iter_queries()The 'blob' columnPinecone dataset ship with a blob column which is inteneded to be used for storing additional data that is not part of the dataset schema. however, it is sometime useful to store additional data in the dataset, for example, a document text. We added a utility function to move data from the blob column to the metadata column. This is useful for example when upserting a dataset to an index and want to use the metadata to store text data.frompinecone_datasetsimportimport_documents_keys_from_blob_to_metadatanew_dataset=import_documents_keys_from_blob_to_metadata(dataset,keys=["text"])Usage savingyou can save your dataset to a catalog managed by you or to a local path or a remote path (GCS or S3).Saving to CatalogTo set you own catalog endpoint, set the environment variableDATASETS_CATALOG_BASEPATHto your bucket. Note that pinecone uses the default authentication method for the storage type (gcsfs for GCS and s3fs for S3).After this environment variable is set you can save your dataset to the catalog using thesavefunctionfrompinecone_datasetsimportDatasetmetadata=DatasetMetadata(**{"name":"my-dataset",...})🚨NOTEDataset name in the metadata must match thedataset_idparameter you pass to the catalog, in this example 'my-dataset'dataset=Dataset.from_pandas(documents,queries,metadata)dataset.to_catalog("my-dataset")Saving to PathYou can save your dataset to a local path or a remote path (GCS or S3). Note that pinecone uses the default authentication method for the storage type (gcsfs for GCS and s3fs for S3).dataset=Dataset.from_pandas(documents,queries,metadata)dataset.to_path("s3://my-bucket/my-subdir/my-dataset")upserting to IndexWhen upserting a Dataset to an Index, only the document data will be upserted to the index. The queries data will be ignored.TODO: add example for API Key adn Environment Variablesds=load_dataset("dataset_name")ds.to_pinecone_index("index_name")# or, if you run in notebook environmentawaitds.to_pinecone_index_async("index_name")theto_pinecone_indexfunction also accepts additional parameters:batch_sizefor controlling the upserting processapi_keyfor passing your API key, otherwise you cankwargs- for passing additional parameters to the index creation processFor developersThis project is using poetry for dependency managemet. supported python version are 3.8+. To start developing, on project root directory run:poetryinstall--withdevTo run test locally runpoetryrunpytest--covpinecone_datasets