Package: gabarit
Gabarit - Templates Data ScienceGabarit provides you with a set of python templates (a.k.a. frameworks) for your Data Science projects. It allows you to generate a code base that includes many features to speed up the production and testing of your AI models. You just have to focus on the core of Data Science.Table of ContentLicenceFrameworksPrerequisitesUsageInstallationKickstart a new projectSetup the new projectSecurity warningEthicsContacts📚 Documentation :https://oss-pole-emploi.github.io/gabaritLicenceThis project is distributed under the GNU AFFERO GENERAL PUBLIC LICENSE V3.0. Please check the LICENSE file.FrameworksAs a team, we strive to help Data Scientists across the board (and ourselves!) build awesome IA projects by speeding up the development process. This repository contains several frameworks allowing any data scientist, IA enthousiast (or developper of any kind, really) to kickstart an IA project from scratch.We hate it when a project is left in the infamous POC shadow valley where nice ideas and clever models are forgotten, thus we tried to pack as much production-ready features as we could in these frameworks.As Hadley Wickhman would say: "you can't do data science in a GUI". We are strong believers that during a data science or IA project, you need to be able to fine tune every nooks and crannies to make the best out of your data.Therefore, these frameworks act as project templates that you can use to generate a code base from nothing (except for a project name). Doing so would allow your fresh and exciting new project to begin with loads of features on which you wouldn't want to focus this early :Built-in models: from the ever useful TF/IDF + SVM to the more recent transformersModel-agnostic save/load/reload : perfect to embed your model behind a web serviceGeneric training/predict scripts to work with your data as soon as possibleDVC & MLFlow integration (you have to configure it to point to your own infrastructures)Streamlit demo tool... and so much more !Three IA Frameworks are available:NLPto tackle classification use cases on textual dataRelies on the Words'n fun module for the preprocessing requirementsSupports :Mono Class / Mono Label classificationMulti Classes / Mono Label classificationMono Class / Multi Labels classificationNumericto tackle classification and regression use cases on numerical dataSupports :RegressionMulti Classes / Mono Label classificationMono Class / Multi Labels classificationComputer Visionto tackle classification use cases on imagesSupportsMono Class / Mono Label classificationMulti Classes / Mono Label classificationArea of interest detectionIn addition we provide anAPI Frameworkthat can be used to expose a gabarit model or one of your own.PrerequisitesTo use these frameworks, you should already have python >= 3.7 installed. Note that this project started in python 3.7 but is now tested with python 3.8.
Obviously any prior knowledge of the holy trinity of python ML modules (pandas, sklearn, numpy) alongside Deep Learning frameworks (torch & tensorflow/keras) would be incredibly useful.UsageInstallationWe packaged this project such that it can be directly installed from PyPI :pip install gabarit.However, it is not really necessary as this just intalls Jinja2==3.0.3 and adds some entry points. Basically you can manually install Jinja2pip install Jinja2==3.0.3and you'll be able to generate new projects by calling thegenerate_XXX_project.pyfiles individually.In the following we consider that you installed the project through pip, which enables entry points. Each entry point refers to a corresponding package generation file.Kickstart a new projectEach individual framework has agenerate_XXX_projectentry point that creates a new project code base.
They take several parameters as input :-nor--name: Name of the package/project (lowercase, no whitespace)-por--path: Path (Absolute or relative) where to create the main directory of the project-cor--config: Path (Absolute or relative) to a .ini configuration file.An default configuration file is given alongside each project. (default_config.ini).
It usually contains stuff like default encoding, default separator for .csv files, pip proxy settings, etc.--uploador--upload_intructions: Path (Absolute or relative) to a file that contains a list of instructions to upload a trained model to your favorite storage solution.--dvcor--dvc_config: Path (Absolute or relative) to a DVC configuration file. If not provided, DVC won't be used.Example :generate_nlp_project -n my_awesome_package -p my_new_project_path -c my_configuration.ini --upload my_instructions.md --dvc dvc_configSetup the new project(Optionnal) We strongly advise to create a python virtual envpip install virtualenvpython -m venv my_awesome_venvcd my_awesome_venv/Scripts/ && activate(windows) orsource my_awesome_venv/bin/activate(linux)Requirements :pip install --no-cache-dir -r requirements.txtSetup the project (in develop mode) :python setup.py developIf themaketool is available, you can use the features provided inMakefile:create-virtualenvandinit-local-envSecurity warningGabarit relies on a number of open source packages and therefore may carry on their potential security vulnerabilities. Our philosophy is to be as transparent as possible, which is why we are actively monitoring the dependabot analysis. In order to limit these vulnerabilities, we are in the regular process of upgrading these packages as soon as we can.
Notice that some packages (namely torch and tensorflow) might lag a few versions behind the actual up to date version due to compatibility issues with CUDA and our own infrastructure.However, we remind you to be vigilant about the security vulnerabilities of the code and models that you will produce with these frameworks. It is your responsibility to ensure that the final product matches the security standards of your organization.EthicsPôle emploi intends to include the development and use of artificial intelligence algorithms and solutions in a sustainable and ethical approach. As such, Pôle emploi has adopted an ethical charter, resulting from collaborative and consultative work. The objective is to guarantee a framework of trust, respectful of the values of Pôle emploi, and to minimize the risks associated with the deployment of these technologies.The pdf file is located inpole-emploi.org:PDF - Ethics charter - Pôle emploiContactsIf you have any question/enquiry feel free to drop us an email :drosdadsopportunitesia.00620@pole-emploi.frGaëlle JOUIS   - Data ScientistGautier SOLARD - Data ScientistNicolas TOUZOT - Product Manager