Package: mlcroissant
mlcroissant 🥐Discovermlcroissant 🥐with thisintroduction tutorial in Google Colab.Python requirementsPython version >= 3.10.If you do not have a Python environment:python3-mvenv~/py3source~/py3/bin/activateInstallpython-mpipinstall".[dev]"The command can fail, for example, due to missing dependencies, e.g.:Failedtobuildpygraphviz
ERROR:Couldnotbuildwheelsforpygraphviz,whichisrequiredtoinstallpyproject.toml-basedprojectsThis can be fixed by runningsudoapt-getinstallpython3-devgraphvizlibgraphviz-devpkg-configConda installationConda can help create a consistent environment.
It can also be useful to install packages without root access.
To use Conda, run:condacreate--namecroissantpython=3.10-y
condaactivatecroissant
condainstallgraphviz
python3-mpipinstall".[dev]"Verify/load a Croissant datasetmlcroissantvalidate--jsonld../../datasets/titanic/metadata.jsonThe command:Exits with 0, printsDoneand displays encountered warnings, when no error was found in the file.Exits with 1 and displays all encountered errors/warnings, otherwise.Similarly, you can generate a dataset by launching:mlcroissantload\--jsonld../../datasets/titanic/metadata.json\--record_setpassengers\--num_records10Loading adistributionviagit+httpsIf theencodingFormatof adistributionisgit+https, please provide the username and password by setting theCROISSANT_GIT_USERNAMEandCROISSANT_GIT_PASSWORDenvironment variables. These will be used to construct the authentication necessary to load the distribution.Note that, for datasets hosted on HuggingFace,CROISSANT_GIT_USERNAMEandCROISSANT_GIT_PASSWORDshould correspond respectively to your HuggingFace's username and User Access Token. User Access Tokens can be generated followingthis guide.Loading adistributionvia HTTP with Basic AuthIf thecontentUrlof adistributionrequires authentication via Basic Auth, please provide the username and password by setting theCROISSANT_BASIC_AUTH_USERNAMEandCROISSANT_BASIC_AUTH_PASSWORDenvironment variables. These will be used to construct the authentication necessary to load the distribution.Programmatically build JSON-LD filesYou can programmatically build Croissant JSON-LD files using the Python API.importmlcroissantasmlcmetadata=mlc.nodes.Metadata(name="...",)metadata.to_json()# this returns the JSON-LD file.Run testsAll tests can be run from the Makefile:maketestsNote thatgit lfsshould be installed to successfully pass all tests:gitlfsinstallDesignThe most important modules in the library are:mlcroissant/_src/structure_graphis responsible for thestatic analysisof the Croissant files. We convert Croissant files to a Python representation called "structure graph" (usingNetworkX). In the process, we catch any static analysis issues (e.g., a missing mandatory field or a logic problem in the file).mlcroissant/_src/operation_graphis responsible for thedynamic analysisof the Croissant files (i.e., actually loading the dataset by yielding examples). We convert the structure graph into an "operation graph". Operations are the unit transformations that allow to build the dataset (likeDownload,Extract, etc).Other important modules are:mlcroissant/_src/coredefines all needed core internals. For instance,Issuesare a way to track errors and warning during the analysis of Croissant files.mlcroissant/__init__.pydeclares the public API withmlcroissant.Dataset.For the full design, refer to thedesign docfor an overview of the implementation.Caching. By default, all downloaded/extracted files are cached in~/.cache/croissant, but you can overwrite this by setting the environment variable$CROISSANT_CACHE.ContributeAll contributions are welcome! We even havegood first issuesto start in the project. Refer to theGitHub projectfor more detailed user stories and read above how the repo isdesigned.An easy way to contribute tomlcroissantis using Croissant's configuredcodespaces.
To start a codespace:On Croissant's mainrepo page, click on the<Code>button and select theCodespacestab. You can start a new codespace by clicking on the+sign on the left side of the tab. By default, the codespace will start on Croissant'smainbranch, unless you select otherwise from the branches drop-down menu on the left side.While building the environment, your codespaces will install allmlcroissant's required dependencies - so that you can start coding right away! Of course, you canfurther personalizeyour codespace.To start contributing to Croissant:Create a new branch from theTerminaltab in the bottom panel of your codespace withgit checkout -b feature/my-awesome-new-featureYou can create new commits, and run most git commands from theSource Controltab in the left panel of your codespace. Alternatively, use theTerminalin the bottom panel of your codespace.Iterate on your code until all tests are green (you can run tests withmake pytestor form theTeststab in the left panel of your codespace).Open a pull request (PR) with the main branch ofhttps://github.com/mlcommons/croissant, and ask for feedback!Alternatively, you can contribute tomlcroissantusing the "classic" GitHub workflow:DebugYou can debug the validation of the file using the--debugflag:mlcroissantvalidate--jsonld../../datasets/titanic/metadata.json--debugThis will:print extra information, like the generated nodes;save the generated structure graph to a folder indicated in the logs.Publishing packagesTo publish a package,Bump the version incroissant/python/mlcroissant/pyproject.toml.Publish a release in GitHub. The workflow scriptpython-publish.ymlwill trigger and publish the package to PyPI.