Package: dsdl
OpenDataLab WebsiteEnglish |简体中文📘Documentation|IntroductionDatais the cornerstone of artificial intelligence. The efficiency of data acquisition, exchange, and application directly impacts the advances in technologies and applications. Over the long history of AI, a vast quantity of data sets have been developed and distributed. However, these datasets are defined in very different forms, which incurs significant overhead when it comes to exchange, integration, and utilization -- it is often the case that one needs to develop a new customized tool or script in order to incorporate a new dataset into a workflow.To overcome such difficulties, we developData Set Description Language (DSDL).Major featuresThe design ofDSDLis driven by three goals, namelygeneric,portable,extensible. We refer to these three goals together asGPE.GenericThis language aims to provide a unified representation standard for data in multiple fields of artificial intelligence, rather than being designed for a single field or task. It should be able to express data sets with different modalities and structures in a consistent format.PortableWrite once, distribute everywhere. Dataset descriptions can be widely distributed and exchanged, and used in different environments without modification of the source files. The achievement of this goal is crucial for creating an open and thriving ecosystem. To this end, we need to carefully examine the details of the design, and remove unnecessary dependencies on specific assumptions about the underlying facilities or organizations.ExtensibleOne should be able to extend the boundary of expression without modifying the core standard. For a programming language such as C++ or Python, its application boundaries can be significantly extended by libraries or packages, while the core language remains stable over a long period. Such libraries and packages form a rich ecosystem, making the language stay alive for a very long time.InstallationCase ainstall it with pippipinstalldsdlCase binstall it from sourcegitclonehttps://github.com/opendatalab/dsdl.gitcddsdl
pythonsetup.pyinstallGet StartedUse dsdl parser to deserialize the Yaml file to Python codecd<dsdlprojectdirectory>
dsdlparse--yamldemo/coco_demo.yamlModify the configuration & set the directory of media in datasetCreate a configuration fileconfig.pywith the following contents（for now dsdl only reading from aliyun oss or local is supported）：local=dict(type="LocalFileReader",working_dir="demo",)ali_oss=dict(type="AliOSSFileReader",access_key_secret="your secret key of aliyun oss",endpoint="your endpoint of aliyun oss",access_key_id="your access key of aliyun oss",bucket_name="your bucket name of aliyun oss",working_dir="the relative path of your media dir in the bucket")Inconfig.py, the configuration of how to read the media in a dataset is defined. One should specify the arguments depending on from where to read the media：read from local：working_dirfield inlocalshould be specified (the directory of local media)read from aliyun oss： all the field inali_ossshould be specified (includingaccess_key_secret,endpoint,access_key_id,bucket_name,working_dir)Visualize samplesdsdlview-y<yaml-name>.yaml-c<config.py>-lali-oss-n5-r-v-fLabelBBoxThe description of each argument is shown below:simplified  argumentargumentdescription-y--yamlThe path of dsdl yaml file.-c--configThe path of  location configuration file.-l--locationlocalorali-oss，which means read media from local or aliyun oss.-n--numThe number of samples to be visualized.-r--randomWhether to load the samples in a random order.-v--visualizeWhether to visualize the samples or just print the information in console.-f--fieldThe field type to visualize, e.g.-f BBoxmeans show the bounding box in samples,-f Labelmeans show the labels of a sample in the console . One can specify multiple field types simultaneously, such as-f Label BBox.-t--taskThe task you are working on, for example,-t detectionis equivalent to-f Label BBox Polygon.CitationIf you find this project useful in your research, please consider cite:@misc{dsdl2022,title={{DSDL}: Data Set Description Language},author={DSDL Contributors},howpublished={\url{https://github.com/opendatalab/dsdl}},year={2022}}LicenseDSDLis released under theApache 2.0 license.AcknowledgementField & Model Design inspired byDjango ORMandjsonmodels