Package: cgnal-core
CGnal coreA Python library defining data structures optimized for machine learning pipelinesWhat is it ?cgnal-coreis a Python package with modular design that provides powerful abstractions to build data
ingestion pipelines and run end to end machine learning pipelines.
The library offers lightweight object-oriented interface to MongoDB as well as Pandas based data structures.
The aim of the library is to provide extensive support for developing machine learning based applications
with a focus on practicing clean code and modular design.FeaturesSome cool features that we are proud to mention are:Data layersArchiver: Offers an object-oriented design to perform ETL on Mongodb collections as well as Pandas DataFrames.DAO: Data Access Object to allow archivers to serialize domain objects into the proper persistence layer support
object (e.g. in the case of MongoDB, a DAO serializes a domain object into a MongoDB document) and to parse objects
retrieved from the given persistence layer in the correct representation in our framework (e.g. a text will be parsed in
a Document while tabular data will be parsed in a pandas DataFrame).Database: Object representing a relational databaseTable: Object representing a table of a relational databaseData ModelOffers the following data structures:Document : Data structure specifically designed to work with NLP applications that parses a json-like document
into a couple of uuid and dictionary of information.Sample : Data structure representing an observation (a.k.a. sample) as used in machine learning applicationsMultiFeatureSample : Data structure representing an observation defined by a nested list of arrays.Dataset : Data structure designed to be used specifically for machine learning applications representing a collection
of samples.InstallationFrom pypi serverpip install cgnal-coreFrom sourcegit clone https://github.com/CGnal/cgnal-core
cd cgnal-core
make installTestsmake testsChecksTo run predefined checks (unit-tests, linting checks, formatting checks and static typing checks):make checksExamplesData LayersCreating a Database of Table objectsimportpandasaspdfromcgnal.core.data.layer.pandas.databasesimportDatabase# sample dfdf1=pd.DataFrame([[1,2,3],[6,5,4]],columns=['a','b','c'])# creating a databasedb=Database('/path/to/db')table1=db.table('df1')# write table to pathtable1.write(df1)# get pathtable1.filename# convert to pandas dataframetable1.to_df()# get table from databasedb.__getitem__('df1')Using an Archiver with Dao objectsfromcgnal.core.data.layer.pandas.archiversimportCsvArchiverfromcgnal.core.data.layer.pandas.daoimportDataFrameDAO# create a dao objectdao=DataFrameDAO()# create a csv archiverarch=CsvArchiver('/path/to/csvfile.csv',dao)# get pandas dataframearch.data# retrieve a single document objectdoc=next(arch.retrieve())# retrieve a list of document objectsdocs=[iforiinarch.retrieve()]# retrieve a document by it's idarch.retrieveById(doc.uuid)# archive a single documentdoc=next(self.a.retrieve())# update column_name field of the document with the given valuedoc.data.update({'column_name':value})# archive the documentarch.archiveOne(doc)# archive list of documentsa.archiveMany([doc,doc])# get a document object as a pandas seriesarch.dao.get(doc)Data ModelCreating a PandasDataset objectimportpandasaspdfromcgnal.core.data.model.mlimportPandasDatasetdataset=PandasDataset(features=pd.concat([pd.Series([1,np.nan,2,3],name="feat1"),pd.Series([1,2,3,4],name="feat2")],axis=1),labels=pd.Series([0,0,0,1],name="Label"))# access features as a pandas dataframedataset.features# access labels as pandas dataframedataset.labels# access features as a python dictionarydataset.getFeaturesAs('dict')# access features as numpy arraydataset.getFeaturesAs('array')# indexing operations# access features and labels at the given index as a pandas dataframedataset.loc(2).featuresdataset.loc(2).labelsCreating a PandasTimeIndexedDataset objectimportpandasaspdfromcgnal.core.data.model.mlimportPandasTimeIndexedDatasetdateStr=[str(x)forxinpd.date_range('2010-01-01','2010-01-04')]dataset=PandasTimeIndexedDataset(features=pd.concat([pd.Series([1,np.nan,2,3],index=dateStr,name="feat1"),pd.Series([1,2,3,4],index=dateStr,name="feat2")],axis=1))How to contribute ?We are very much willing to welcome any kind of contribution whether it is bug report, bug fixes, contributions to the
existing codebase or improving the documentation.Where to start ?Please look at theGithub issues tabto start working on open
issuesContributing to cgnal-corePlease make sure the general guidelines for contributing to the code base are respectedForkthe cgnal-core repository.Create/choose an issue to work on in theGithub issues page.Create a new branchto work on the issue.Commit your changes and run the tests to make sure the changes do not break any test.Open a Pull Request on Github referencing the issue.Once the PR is approved, the maintainers will merge it on the main branch.