Package: py4ai-data
py4ai dataA Python library defining data structures optimized for machine learning pipelinesWhat is it ?py4ai-datais a Python package with modular design that provides powerful abstractions to build data
ingestion pipelines and run end to end machine learning pipelines.
The library offers lightweight object-oriented interface to MongoDB as well as Pandas based data structures.
The aim of the library is to provide extensive support for developing machine learning based applications
with a focus on practicing clean code and modular design.
The detailed documentation for all its features is availablehere.FeaturesSome cool features that we are proud to mention are:Data layersThe Data Layer abstractions are designed to decouple the business layers from
the detail of the persistence layer implementation.
We currently implemented a few data layers based on some common abstractions:Repository: class representing the access point to data. Specifies the methods to read/write data from/to persistence layer. It must be instantiated by passing it a Serializer, to specify how to read/write data from/to persistence layer to/from memory.Serializer: class that specifies how to read/write data from/to persistence layer to/from memory.Criteria: class implementing query method to filter data retrieved from repository.
The data layer currently implemented are:file-system: access generic data stored in the file system as generic files, handling them in memory as generic indexed objects.pandas: access tabular data stored in the file system as csv files, handling them in memory as pandas objects.mongo: access data stored in MongoDB. Serializers for this layer should be implemented by the user according to its specific needs.sqlalchemy: access data stored in a database handled with sqlalchemy ORM. Serializers for this layer should be implemented by the user according to its specific needs.Data ModelOffers the following data structures:Document : Data structure specifically designed to work with NLP applications that parses a json-like document
into a couple of uuid and dictionary of information.Sample : Data structure representing an observation (a.k.a. sample) as used in machine learning applicationsMultiFeatureSample : Data structure representing an observation defined by a nested list of arrays.Dataset : Data structure designed to be used specifically for machine learning applications representing a collection
of samples.InstallationFrom pypi serverpip install py4ai-dataFrom sourcegit clone https://github.com/NicolaDonelli/py4ai-data
cd py4ai-data
make installTestsmake testsChecksTo run predefined checks (unit-tests, linting checks, formatting checks and static typing checks):make checksExamplesData LayersThe Data Layer abstractions are designed to decouple the business layers from
the detail of the persistence layer implementation. The basic abstraction that will
make this possible is theRepository.As an example, imagine to have a domain business objectEntitythat associates some string data to an integer id.frompydanticimportBaseModelclassEntity(BaseModel):my_id:intmy_data:strTo start with, imagine we want to use csv files store on disk as a persistence
layer. To do so, we will use theCsvRepositorythat uses pandas DataFrames stored
in memory and written to the disk as csv. Thus, we need to write the business logic
to serialize theEntityinto a row of the pandas DataFrame, i.e. a pandas Series:importpandasaspdfrompy4ai.data.layer.common.serialiazerimportDataSerializerclassEntitySerializer(DataSerializer[int,int,Entity,pd.Series]):defto_object(self,entity:Entity)->pd.Series:returnpd.Series(entity.dict())defto_entity(self,document:pd.Series)->Entity:returnEntity(**document)defto_object_key(self,key:int)->int:returnkeydefget_key(self,entity:Entity)->int:returnentity.my_idWe can now instantiate the repository class that has all the methods for
reading and writing objects from the persistence layer.frompy4ai.data.layer.pandas.repositoryimportCsvRepositoryrepo=CsvRepository(filename,EntitySerializer())entity=Entity(my_id=1234,my_data="Important data")# This will create the entity in the persistence layerawaitrepo.create(entity)# Retrieving the entityretrieved=repo.retrieve(key=1234)# Retrieving all entitiesall_entities=repo.list()Imagine now that, given the data increase in size, we now would like to change the
persistence layer with a proper backend into something more structured and scalable, such
as a NoSQL document-based data platform, such as MongoDB. We only need to create
a new business logic to serialize/deserialize our class into a json (represented
in python by a dictionary):frombsonimportObjectIdfrompy4ai.data.layer.mongo.serializerimportcreate_mongo_idfrompy4ai.data.layer.common.serialiazerimportDataSerializerclassMongoDataSerializer(DataSerializer[int,ObjectId,Entity,dict]):defget_key(self,entity:Entity)->int:returnentity.my_iddefto_object(self,entity:Entity)->dict:doc=entity.dict()doc["_id"]=self.to_object_key(self.get_key(entity))returndocdefto_entity(self,document:dict)->Entity:returnEntity(**document)defto_object_key(self,key:int)->ObjectId:# This converts a string into an hash compatible with MongoDB formatreturncreate_mongo_id(str(key))A new repository based on the MongoDB persistence layer can now be created usingfrompy4ai.data.layer.mongo.repositoryimportMongoRepositoryrepo=MongoRepository(collection,MongoDataSerializer())This repository is compatible with the previous and can be used in place of the
previous one, having the same signatures.Abstracting Data QueryingTheRepositoryabstraction also allow to retrieve data based on certain queries/filters:entities=repo.retrieve_by_query(criteria)However, the format of the query also depends on the type of the persistence layer and
more specifically on how the data are organized. Therefore, in order to abstract
and decouple the notion of the underlying persistence layer, we need to define a general
class containing the possible queries for a certain database:fromtypingimportGenericfromabcimportABC,abstractmethodfrompy4ai.data.layer.common.criteriaimportSearchCriteriaclassEntityQueryFactory(ABC,Generic[Q]):@abstractmethoddefby_id(self,id:int)->SearchCriteria[Q]:...When considering a particular persistence layer, the querying business logic
needs to be specifiedfrompy4ai.data.layer.mongo.criteriaimportMongoSearchCriteriaclassMongoQueryFactory(EntityQueryFactory[Dict[str,Any]]):defby_id(self,my_id:int)->MongoSearchCriteria:returnMongoSearchCriteria({"my_id":my_id})criteria=MongoQueryFactory()entities=repo.retrieve_by_query(criteria.by_id(1234))Note thatSearchCriteriacan be also joined using logical operators:entities=repo.retrieve_by_query(criteria.by_id(1234)&criteria.by_id(1235))entities=repo.retrieve_by_query(criteria.by_id(1234)|criteria.by_id(1235))Data ModelThedata.modelmodule contains data structures commonly used.
The most relevant structures defined in thecoresubmodule areCachedIterableandLazyIterablethat represent iterable objects that are either completely stored in memory or accessed lazily. In
the same submodule we also defineRangeandCompositeRangethat represent time and numeric ranges,
either continuous or not, with methods to compare two instances checking for overlaps or performing
range union by sum.Themlsubmodule on the other hand defines some data structures that are of common use in
Machine Learning applications.
The basic structure we define is theSample, which is a simple serializable object characterized
by a set of features, an optional set of labels and an optional name identifier:classSample(DillSerialization,Generic[FeatType,LabType]):"""Base class for representing a sample/observation."""def__init__(self,features:FeatType,label:Optional[LabType]=None,name:Optional[Union[int,str,Any]]=None,)->None:"""Return an object representing a single sample of a training or test set.:param features: features of the sample:param label: labels of the sample (optional):param name: id of the sample (optional)"""self.features:FeatType=featuresself.label:Optional[LabType]=labelself.name:Optional[Union[str,int,Any]]=nameWe also define a specialization of this class, calledMultiFeatureSample,
thought for Deep Learning applications, for which the features attribute is a list of numpy arrays.
The most used objects defined in this module are the datasets, which represents collections of Samples
in various fashions. All the datasets share some common method allowing to retrieve the type of the Samples,
to check the names of the samples, to retrieve features and labels in a form to be chosen amongpd.DataFrame,np.ndarray,dict,list, or a generator of samples. In particular, we defined:LazyDataset: a collection with lazy access to samplesCachedDataset: a collection of samples fully stored in memoryPandasDataset: a collection of samples, fully stored in memory, where features and labels are pd.DataFramesPandasTimeIndexedDataset: a collection of samples, fully stored in memory, where features and labels are pd.DataFrames with time indexCreating a PandasDataset objectimportpandasaspdimportnumpyasnpfrompy4ai.data.model.mlimportPandasDatasetdataset=PandasDataset(features=pd.concat([pd.Series([1,np.nan,2,3],name="feat1"),pd.Series([1,2,3,4],name="feat2")],axis=1),labels=pd.Series([0,0,0,1],name="Label"))# access features as a pandas dataframeprint(dataset.features.head())# access labels as pandas dataframeprint(dataset.labels.head())# access features as a python dictionarydataset.getFeaturesAs('dict')# access features as numpy arraydataset.getFeaturesAs('array')# indexing operations# access features and labels at the given index as a pandas dataframeprint(dataset.loc([2]).features.head())print(dataset.loc([2]).labels.head())Creating a PandasTimeIndexedDataset objectimportpandasaspdimportnumpyasnpfrompy4ai.data.model.mlimportPandasTimeIndexedDatasetdateStr=[str(x)forxinpd.date_range('2010-01-01','2010-01-04')]dataset=PandasTimeIndexedDataset(features=pd.concat([pd.Series([1,np.nan,2,3],index=dateStr,name="feat1"),pd.Series([1,2,3,4],index=dateStr,name="feat2")],axis=1))How to contribute ?We are very much willing to welcome any kind of contribution whether it is bug report, bug fixes, contributions to the
existing codebase or improving the documentation.Where to start ?Please look at theGithub issues tabto start working on open
issuesContributing to py4ai-dataPlease make sure the general guidelines for contributing to the code base are respectedForkthe py4ai-data repository.Create/choose an issue to work on in theGithub issues page.Create a new branchto work on the issue.Commit your changes and run the tests to make sure the changes do not break any test.Open a Pull Request on Github referencing the issue.Once the PR is approved, the maintainers will merge it on the main branch.