Package: cleanlab-cli
cleanlab-cliCommand line interface for all things Cleanlab Studio.This currently supports generatingdataset schema, uploadingdatasetsinto Cleanlab Studio, and downloading cleansets from Cleanlab Studio.InstallationYou can install the Cleanlab Studio CLIfrom PyPIwith:pipinstallcleanlab-cliIf you already have the CLI installed and wish to upgrade to the latest version, run:pipinstall--upgradecleanlab-cliWorkflowUploading datasets to Cleanlab Studio is a two-step process.We generate a schema describing the dataset and itsdata and feature
types, which is verified by the user.Based on this schema, the dataset is parsed and uploaded to Cleanlab Studio.Upload a datasetTo upload a dataset without
firstgenerating a schema(i.e. Cleanlab will
suggest one for you):cleanlab dataset upload -f [dataset filepath]You will be asked to"Specify your dataset modality (text, tabular):".Entertextto only find label errors based on a single column of text in your dataset.Entertabularto find data and label issues based on any subset of the column features.To upload a dataset with a schema:cleanlab dataset upload -f [dataset filepath] -s [schema filepath]Toresume uploadinga dataset whose upload was interrupted:cleanlab dataset upload -f [dataset filepath] --id [dataset ID]A dataset ID is generated and printed to the terminal the first time the dataset is uploaded. It can also be accessed by
visitinghttps://app.cleanlab.ai/datasetsand selecting 'Resume' for the relevant dataset.Generate dataset schemaTo generate a dataset schema (prior to uploading your dataset):cleanlab dataset schema generate -f [dataset filepath]ForId column:, please enter the string name of of the column in your dataset that contains the id of each row.ForModality (text, tabular):, please entertextto only find label errors based on a single column of text,
otherwise entertabularto find data and label issues based on any subset of the column features.To validate an existing schema, i.e. check that it is complete, well-formatted, and
hasdata types with sensible feature types:cleanlab dataset schema validate -s [schema filepath]You may then wish to inspect the generated schema to check that the fields and metadata are correct.Download clean labelsTo download clean labels (i.e. labels that have been fixed through the Cleanlab Studio interface):cleanlab cleanset download --id [cleanset ID]To download clean labels and combine them with your local dataset:cleanlab cleanset download --id [cleanset ID] -f [dataset filepath]Commandscleanlab loginauthenticates youAuthenticates you when uploading datasets to Cleanlab Studio. Pass in your API key using--key [API key]. Your API key
can be accessed athttps://app.cleanlab.ai/upload.cleanlab dataset schema generategenerates dataset schemasGenerates a schema based on your dataset. Specify your target dataset with--filepath [dataset filepath]. You will be
prompted to save the generated schema JSON and to specify a save location. This can be specified
using--output [output filepath].cleanlab dataset schema validatevalidates a schema JSON fileValidates a schema JSON file, checking that a schema is complete, well-formatted, and
hasdata types with sensible feature types. Specify your target schema
with--schema [schema filepath].You may also validate an existing schema with respect to a dataset (-d [dataset filepath]), i.e. all previously
mentioned checks and the additional check that all fields in the schema are present in the dataset.cleanlab dataset uploaduploads your datasetUploads your dataset to Cleanlab Studio. Specify your target dataset with--filepath [dataset filepath]. You will be
prompted for further details about the dataset's modality and ID column. These may be supplied to the command
with--modality [modality],--id-column [name of ID column], and you may also specify a custom dataset name
with--name [custom dataset name].After uploading your dataset, you will be prompted to save the list of dataset issues (if any) encountered during the
upload process. These issues include missing IDs, duplicate IDs, missing values, and values whose types do not match the
schema. You may specify the save location with--output [output filepath].cleanlab cleanset downloaddownloads Cleanlab columns from your cleansetCleansets are initialized through the Cleanlab Studio interface. In a cleanset, users can inspect their dataset and
verify their labels. Clean labels are the labels after this set of manual fixes have been applied.This command downloads the clean labels and saves them locally as a .csv, .xls/.xlsx, or .json, with columnsidandclean_label. Include the--filepath [dataset filepath]to combine the clean labels with the original dataset as
a new columnclean_label, which will be outputted to--output [output filepath]. Include the--allflag to
includeallCleanlab columns, i.e. issue, label quality, suggested label, clean label, instead of only the clean
label column.Dataset formatCleanlab currently only supports text and tabular dataset modalities.
(If your dataset contains both text and tabular data, treat it as tabular.)
The accepted dataset file types are:.csv,.json, and.xls/.xlsx.Below are some examples of how to format your dataset depending on modality and file type.Every dataset must have anID column(i.e. a column containing identifiers that uniquely identify each row) and alabel column(for the prediction task).Apart from the reserved column name:clean_label, You are free to name the columns in your dataset in any way you
want.Tabular.csv, .xls/.xlsxflower_idwidthlengthcolorspeciesflower_0143redroseflower_0272whitelily.json{"rows":[{"flower_id":"flower_01","width":4,"length":3,"color":"red","species":"rose"},{"flower_id":"flower_02","width":7,"length":2,"color":"white","species":"lily"}]}Text.csv, .xls/.xlsxreview_idreviewsentimentreview_1The sales rep was fantastic!positivereview_2He was a bit wishy-washy.negative.json{"rows":[{"review_id":"review_1","review":"The sales rep was fantastic!","label":"positive"},{"review_id":"review_2","review":"He was a bit wishy-washy.","label":"negative"}]}SchemaTo specify the column types in your dataset, create a JSON file namedschema.json. We recommend
usingcleanlab dataset schema generateto generate an initial schema and editing from there.Your schema file should be formatted as follows:{
  "metadata": {
    "id_column": "tweet_id",
    "modality": "text",
    "name": "Tweets.csv"
  },
  "fields": {
    "tweet_id": {
      "data_type": "string",
      "feature_type": "identifier"
    },
    "sentiment": {
      "data_type": "string",
      "feature_type": "categorical"
    },
    "sentiment_confidence": {
      "data_type": "float",
      "feature_type": "numeric"
    },
    "retweet_count": {
      "data_type": "integer",
      "feature_type": "numeric"
    },
    "text": {
      "data_type": "string",
      "feature_type": "text"
    },
    "tweet_created": {
      "data_type": "boolean",
      "feature_type": "boolean"
    },
    "tweet_created": {
      "data_type": "string",
      "feature_type": "datetime"
    },
  },
  "version": "0.1.12"
}This is the schema of a hypothetical datasetTweets.csvthat contains tweets, where the columntweet_idcontains a
unique identifier for each record. Each column in the dataset is specified underfieldswith its data type and feature
type.Data types and Feature typesData typerefers to the type of the field's values: string, integer, float, or boolean.Note that the integer type isstrict, meaning floats will be rejected. In contrast, the float type islenient,
meaning integers are accepted. Users should select the float type if the field may include float values. Note too that
integers can have categorical and identifier feature types, whereas floats cannot.For booleans, the list of accepted values are: true/false, t/f, yes/no, and 1/0.Feature typerefers to the secondary type of the field, relating to how it is used in a machine learning model, such
as whether it is:a categorical valuea numeric valuea datetime valuea boolean valuetextan identifier — a string / integer that identifies some entitySome feature types can only correspond to specific data types. The list of possible feature types for each data type is
shown belowData typeFeature typestringtext, categorical, datetime, identifierintegercategorical, datetime, identifier, numericfloatdatetime, numericbooleanbooleanThedatetimetype should be used for datetime strings, e.g. "2015-02-24 11:35:52 -0800", and Unix timestamps (which
will be integers or floats). Datetime values must be parsable
bypandas.to_datetime().versionindicates the version of the Cleanlab CLI package version used to generate the schema. The current Cleanlab
schema version is0.1.14.