Package: pytdml
pytdmlpytdmlis a pure python parser and encoder for training datasets based on OGC
Training Data Markup Language for AI standard.InstallationThe package can be installed via pip.RequirementsPython 3 and aboveDependenciesDependencies are listed inrequirements.txt. Dependencies are automatically installed during
pytdml's installation.Installing the PackagepipinstallpytdmlUsageEncodingFrom the command lineThe training dataset can be encoded to TrainingDML-AI JSON format by YAML configuration file with command line.pytdml/yaml_to_tdml.py--config=<YAMLconfigurationfilepath>--output=<OutputTrainingDML-AIJSONfilepath>YAML configuration file schema is described inencoding YAML configuration file schema.Using the API from pythonThe training dataset can also be encoded to TrainingDML-AI JSON format with Python API.frompytdml.ioimportwrite_to_jsonfrompytdml.typeimportEOTrainingDataset,EOTrainingData,EOTask,EODataSource,SceneLabel# generate EO training datasetdataset=EOTrainingDataset(id='...',name='...',description='...',data=[EOTrainingData(id='...',labels=[SceneLabel(label_class='...',data_url='...',date_time='...'),...]),...],version="...",amount_of_training_data=...,created_time="...",updated_time="...",providers=["..."],keywords=["...","..."],tasks=[EOTask(task_type="...",description="...")],data_sources=[EODataSource(id="...",data_type="...",resolution="...")],classes=["...","...","..."],number_of_classes=...,bands=["...","...","..."],image_size="...")# write to jsonwrite_to_json(dataset,"dataset.json")encoding training data from S3# get training data from s3s3_client=pytdml.io.S3Client('s3',"your_server","your_ak","your_sk")td_list=[]bucket_name="my-bucket"obj_list=s3_client.list_objects(Bucket=bucket_name,Prefix="whu_rs19/")forobjinobj_list:td=EOTrainingData(id=obj.split(".")[0],labels=[SceneLabel(label_class=obj.split("/")[1])],data_url=f"s3://{bucket_name}/{quote(obj)}",date_time="2010")td_list.append(td)ParsingThe training dataset described with TrainingDML-AI JSON file can be parsed with python API and transformed to
PyTorch/TensorFlow dataset.Read TrainingDataset object from JSON fileimportpytdmltraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json fileprint("Load training dataset: "+training_dataset.name)print("Number of training samples: "+str(training_dataset.amount_of_training_data))print("Number of classes: "+str(training_dataset.number_of_classes))read training data from s3importpytdml# Initialize S3clients3_client=pytdml.io.S3_reader.S3Client("s3","your_server","your_akey","your_skey")# Load the training datasettraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json fileforitemintraining_dataset.data:path=item.data_urlifpytdml.io.S3_reader.pasrse_s3_path(path):bucket_name,key_name=pytdml.io.S3_reader.parse_s3_path(path)object_data=s3_client.get_object(bucket_name,key_name)# Process the S3 object data (read as PIL Image)withPIL.Image.open(BytesIO(object_data))asimg:# processing....else:print("Invalid S3 path:",path)Transform to PyTorch datasetScene classification datasetimportpytdmlfromtorchvisionimporttransforms# Load the training datasettraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json file# Transform the training datasetclass_map=pytdml.ml.creat_class_map(training_dataset)# create class maptrain_dataset=pytdml.ml.TorchEOImageSceneTD(# create Torch train datasettraining_dataset.data,class_map,transform=transforms.Compose(# transform for the training set[transforms.RandomResizedCrop(size=156,scale=(0.8,1.0)),# random resizetransforms.RandomRotation(degrees=15),# random rotatetransforms.RandomHorizontalFlip(),# random fliptransforms.CenterCrop(size=124),# center croptransforms.ToTensor(),# transform to tensortransforms.Normalize([0.485,0.456,0.406],[0.229,0.224,0.225])# normalize]))Object detection datasetimportpytdml# Load the training datasettraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json file# Transform the training datasetclass_map=pytdml.ml.creat_class_map(training_dataset)# create class maptrain_dataset=pytdml.ml.TorchEOImageObjectTD(# create Torch train datasettraining_dataset.data,class_map,transform=pytdml.ml.BaseTransform([128,128]))Semantic segmentation datasetimportpytdmlfromtorchvisionimporttransforms# Load the training datasettraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json file# Transform the training datasetclass_map=pytdml.ml.creat_class_map(training_dataset)# create class maptrain_dataset=pytdml.ml.TorchEOImageSegmentationTD(# create Torch train datasettraining_dataset.data,class_map,transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225])]))Transform to TensorFlow datasetScene classification datasetimportpytdml# Load the training datasettraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json file# Transform the training datasetclass_map=pytdml.ml.creat_class_map(training_dataset)# create class maptrain_dataset=pytdml.ml.TensorflowEOImageSceneTD(# create TensorFlow train datasettraining_dataset.data,class_map)tf_train_dataset=train_dataset.create_dataset()Object detection datasetimportpytdml# Load the training datasettraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json file# Transform the training datasetclass_map=pytdml.ml.creat_class_map(training_dataset)# create class maptrain_dataset=pytdml.ml.TensorflowEOImageObjectTD(# create TensorFlow train datasettraining_dataset.data,class_map)tf_train_dataset=train_dataset.create_dataset()Semantic segmentation datasetimportpytdml# Load the training datasettraining_dataset=pytdml.io.read_from_json("dataset.json")# read from TDML json file# Transform the training datasetclass_map=pytdml.ml.creat_class_map(training_dataset)# create class maptrain_dataset=pytdml.ml.TensorflowEOImageSegmentationTD(# create TensorFlow train datasettraining_dataset.data,class_map)tf_train_dataset=train_dataset.create_dataset()Image CroppingThe images of training dataset in TrainingDML-AI JSON format can be cropped with command line for preprocessing.pytdml/tdml_image_crop.py--input=<InputoriginalTrainingDML-AUfilepath>--output_json=<OutputresultTrainingDML-AIJSONfilepath>--output_images=<Outputdirofresultcroppedimages>--size=<Cropsizeofimages>