Package: adansons-base
Adansons Base DocumentProduct Concept0. Get Access Key1. Installation2. Configuration2.1 with CLI2.2 Environment Variables3. Tutorial 1: Organize meta data and Create datasetStep 0. prepare sample datasetStep 1. create new projectStep 2. import data filesStep 3. import external metadata filesStep 4. filter and export dataset with CLIStep 5. filter and export dataset with Python SDK4. API Reference4.1 Command Reference4.2 Python ReferenceProduct ConceptAdansons Base is a data management tool that organizes metadata of unstructured data and creates and organizes datasets.It makes dataset creation more effective, helps find essential insights from training results, and improves AI performance.More detail
↓↓↓Mediumhttps://medium.com/@KenichiHiguchi/3-things-you-need-to-deal-with-in-data-management-to-create-best-dataset-781177507fc2Product Pagehttps://adansons.wraptas.site0. Get Access KeyType your email into the form below to join our slack and get the access key.Invitation Form:https://share.hsforms.com/1KG8Hp2kwSjC6fjVwwlklZA8moen1. InstallationAdansons Base contains Command Line Interface (CLI) and Python SDK, and you can install both withpipcommand.pipinstallgit+https://github.com/adansons/baseNote: if you want to use CLI in any directory, you have to install with the python globally installed on your computer.2. Configuration2.1 with CLIwhen you run any Base CLI command for the first time, Base will ask for your access key provided on our slack.then, Base will verify the specified access key was correct.if you don't have an access key, please see0. Get Access Key.this command will show you what projects you havebaselistOutputWelcome to Adansons Base!!

Let's start with your access key provided on our slack.

Please register your access_key: xxxxxxxxxx

Successfully configured as xxxx@yyyy.com

projects
========2.2 Environment Variablesif you don’t want to configure interactively, you can use environment variables for configuration.BASE_USER_IDis used for the identification of users, this is the email address you submitted via our form.exportBASE_ACCESS_KEY=xxxxxxxxxxexportBASE_USER_ID=xxxx@yyyy.com3. Tutorial 1: Organize metadata and Create a datasetlet’s start the Base tutorial with the mnist dataset.Step 0. prepare sample datasetinstall dependencies for download dataset at first.pipinstallpypngthen, download a script for mnist from our Base repositorycurl-sSLhttps://raw.githubusercontent.com/adansons/base/main/download_mnist.py>download_mnist.pyrun the download-mnist script. you can specify any folder for downloading as the last argument(default “~/dataset/mnist”). if you run this command on Windows, please replace it with the windows path like “C:\dataset\mnist”python3./download_mnist.py~/dataset/mnistNote: Base can link the data files if you put them anywhere on the local computer. So if you already downloaded the mnist dataset, you can use itafter downloading, you can see data files in ~/dataset/mnist.~
└── dataset
     └── mnist
          ├── train
          │ 	 ├── 0
          │ 	 │   ├── 1.png
          │ 	 │   ├── ...
          │ 	 │   └── 59987.png
          │ 	 ├── ...
          │ 	 └── 9
          └──	test
                ├── 0
                └── ...Step 1. create a new projectcreate mnist project withbase newcommand.basenewmnistOutputYour Project UID
----------------
abcdefghij0123456789

save Project UID in the local file (~/.base/projects)Base will issue a Project Unique ID and automatically save it in a local file.Step 2. import data filesafter step 0, you have many png image files on the”~/dataset/mnist” directory.let’s upload metadata related to their paths into the mnist project with thebase importcommand.baseimportmnist--directory~/dataset/mnist--extensionpng--parse"{dataType}/{label}/{id}.png"Note: if you changed the download folder, please replace “~/dataset/mnist” in the above command.OutputCheck datafiles...
found 70000 files with png extension.
Success!Step 3. import external metadata filesif you have external metadata files, you can integrate them into the existing project database with the—-external-fileoption.in this time, we usewrongImagesInMNISTTestset.csvpublished on Github by youkaichao.https://github.com/youkaichao/mnist-wrong-testthis is the extra metadata that correct wrong label on the mnist test dataset.you can evaluate your model more strictly and correctly by using these extra metadata with Base.download external CSVcurl-SLhttps://raw.githubusercontent.com/youkaichao/mnist-wrong-test/master/wrongImagesInMNISTTestset.csv>~/Downloads/wrongImagesInMNISTTestset.csvbaseimportmnist--external-file--path~/Downloads/wrongImagesInMNISTTestset.csv-adataType:testOutput1 tables found!
now estimating the rule for table joining...

1 table joining rule was estimated!
Below table joining rule will be applied...

Rule no.1

        key 'index'     ->      connected to 'id' key on exist table
        key 'originalLabel'     ->      connected to 'label' key on exist table
        key 'correction'        ->      newly added

1 tables will be applied
Table 1 sample record:
        {'index': 8, 'originalLabel': 5, 'correction': '-1'}

Do you want to perform table join?
        Base will join tables with that rule described above.

        'y' will be accepted to approve.

        Enter a value: y
Success!Step 4. filter and export dataset with CLInow, we are ready to create a dataset.let’s pick up a part of data files, the label is 0, 1, or 2 for training, from project mnist withbase search <project>command.you can use--conditions <value-only-search>option for magical search filter and--query <key-value-pair-search>option for advanced filter.Note that the--queryoption can only use the value for searching.Be careful that you may get so large output on your console without the-s, --summaryoption.The--queryoption's grammar is below.--query {KeyName} {Operator} {Values}add 1 space between each sectiondon't use space any otherYou can use these operators below in the query option.[operators]== : equal
  != : not equal
  >= : greater than
  <= : less than
  >  : greater
  <  : less
  in : inner list of Values
  not in : not inner list of Values(checksearch docsfor more information).basesearchmnist--conditions"train"--query"label in ['1','2','3']"Note: in the query option, you have to specify each component as a string in the list without space like“[’1’,’2’,’3’]”, when you want to operateinornot inquery.Output18831 files
========
'/home/xxxx/dataset/mnist/train/1/42485.png'
...Note: If you specify no conditions or query, Base will return whole data files.If you want to use the 'OR search' with the--querycommand, please use our Python SDK.Step 5. filter and export dataset with Python SDKin python script, you can filter and export datasets easily and simply withProject classandFiles class. (seeSDK docs)(If you don't have the packages below, please install them by usingpip)pipinstallNumPypillowtorchtorchvisionfrombaseimportProject,DatasetimportnumpyasnpfromPILimportImage# export dataset as you want to useproject=Project("mnist")files=project.files(conditions="train",query=["label in ['1','2','3']"])print(files[0])# this returns path-like `File` object# -> '/home/xxxx/dataset/mnist/0/12909.png'print(files[0].label)# this returns the value of attribute 'lable' of first `File` object# -> '0'# function to load image from path# this is necessary, if you want to use image in your dataset# because base Dataset class doesn't convert path to imagedefpreprocess_func(path):image=Image.open(path)image=image.resize((28,28))image=np.array(image)returnimagedataset=Dataset(files,target_key="label",transform=preprocess_func)# you can also use dataset objects like this.fordata,labelindataset:# data: an image-data. ndarray# label: the label of an image data, like 0passx_train,x_test,y_train,y_test=dataset.train_test_split(split_rate=0.2)# or use with torchimporttorchimporttorchvision.transformsastransformsfromPILimportImagedefpreprocess_func(path):image=transforms.ToTensor()(transforms.Resize((28,28))(Image.open(path)))returnimagedataset=Dataset(files,target_key="label",transform=preprocess_func)loader=torch.utils.data.DataLoader(dataset,batch_size=32,shuffle=True)finally, let’s try one of the most characteristic use cases on Adansons Base.in the external file, you imported in step.3, some mnist test data files are annotated as“-1”in the correction column. this means that it is difficult to classify that files even for a human.so, you should exclude that files from your dataset to evaluate your AI models more properly.# you can exclude files which have "-1" on "correction" with below codeeval_files=project.files(conditions="test",query=["correction != -1"])print(len(eval_files))# this returns the number of files matched with requested conditions or query# -> 9963eval_dataset=Dataset(eval_files,target_key="label",transform=preprocess_func)4. API Reference4.1 Command ReferenceCommand Reference4.2 Python ReferencePython Reference