Package: dsmanager
Data Science Manager 👨‍💻Data Science tools to ease access and use of data and modelsInstallThe easiest way to install scikit-learn is usingpip:pipinstalldsmanagerorpoetrypoetryadddsmanagerorcondacondainstalldsmanagerMultiple sub dependencies are available depending on the needs:pipinstalldsmanager[sharepoint]# Add Sharepoint source handlingpipinstalldsmanager[salesforce]# Add SalesForce source handlingpipinstalldsmanager[kaggle]# Add Kaggle source handlingpipinstalldsmanager[snowflake]# Add Snowflkae source handlingpipinstalldsmanager[mysql]# Add MySQL source handlingpipinstalldsmanager[pgsql]# Add PostgreSQL source handlingpipinstalldsmanager[all_sources]# All the supported sourcesUsageThe DS Manager has 3 main components:ADataManagercomponentAControllercomponentAModelcomponentDataManagerThe DataManager allows to manage different types of data sources among which one can mention:File (File locally or online)Http (Http requests)Ftp (Ftp hosted files)Sql (Sql database tables)Sharepoint (Microsoft OneDrive files)SalesForce (SalesForce classes)Kaggle (Kaggle datasets)The first step to use the DataManager is to instance it with a metadata path.fromdsmanagerimportDataManagerdm=DataManager("data/metadata.json")The metadata file is generated if it does not exist and it consist of a dict of sources following this schema:{"SOURCE_NAME":{"source_type":"name_of_the_source","args":{}}}Each source has asource_typecorresponding to the name of the source. You can access this list with this command:DataManager().datasourcesEach of these data sources has its own read and write schemas because of its own parameters requierements. You can also add additional arguments which are not required with the parameterargs.You can obtain the schemas for a specific datasource with the following commands:source_name="file"DataManager().datasources[source_name].read_schema#use write_schema for the output sources.Output:{"source_type":"file","path":"local_path | online_uri","file_type":"csv | excel | text | json | ...","encoding":"utf-8","args":{"pandas_read_file_argument_keyword":"value_for_this_argument"}}DevelopmentSource codeYou can check the latest sources with the command:gitclonehttps://gitlab.com/bigrayou/dsmanagerTestingAfter installation, you can launch the test suite from outside the dsmanager directory (you will need to have pytest >= 7.1.3 installed):pytest-vDependenciesThe DSManager requires:aiohttp >=3.8.3cryptography 38.0.4dash >=2.7.1,<3.0.0llvmlite >=0.39.1,<0.40.0nest-asyncio >=1.5.6,<2.0.0numba >=0.56.4,<0.57.0numexpr >=2.8.4,<3.0.0numpy >=1.23.3,<2.0.0openpyxl >=3.0.10,<4.0.0optuna >=3.0.5,<4.0.0pandas >=1.5.0,<2.0.0paramiko >=2.12.0,<3.0.0pickle-mixin >=1.0.2,<2.0.0python-dotenv >=0.21.0,<0.22.0requests >=2.28.1,<3.0.0scikit-learn >=1.2.0,<2.0.0setuptools >=65.6.3,<66.0.0shap >=0.41.0,<0.42.0sqlalchemy >=1.4.45,<2.0.0sweetviz >=2.1.4,<3.0.0tqdm >=4.64.1,<5.0.0Optionnaly, the DSManager could require:azure-common >=1.1.28,<2.0.0azure-storage-blob >=12.14.1,<13.0.0azure-storage-common >=2.1.0,<3.0.0kaggle >=1.5.12,<2.0.0mysqlclient >=2.1.1,<3.0.0psycopg2-binary >=2.9.5,<3.0.0shareplum >=0.5.1,<0.6.0simple-salesforce >=1.12.2,<2.0.0snowflake-connector-python >=2.9.0,<3.0.0snowflake-sqlalchemy >=1.4.4,<2.0.0Author👤Rayane AmroucheGithub:@AARayaneGitlab:@Bigrayou