DCAI-CLUD: a data-centric framework for the construction of land-use datasets
Creators: name: Yao, Yao
Publisher: name: figshare
Year: 2024
Subjects: classificationCode: 370499
subject: Geoinformatics not elsewhere classified

description: <b>The DCAI-CLUD Content</b>This project is linked to a submission to International Journal of Geographical Information Science, entitled “<b>DCAI-CLUD: a data-centric framework for the construction of land-use datasets</b>”, which proposed a framework for optimizing the land-use dataset construction process based on Data-centric artificial intelligence.<b>1. </b><b>Overview</b>Readers can follow the steps below to recrate the results in the paper:For the sample filtering method based on size and location of parcels, we provide the sampled data, and readers can reproduce our work using their own familiar tools such as SQL or Excel according to the method introduced in the paper;For the land use classification model used in the human-computer collaborative data labeling method, we provide a simplified version of BDF-Net and sample data.For the statistics of the labelers in the paper, we provide all the data so that readers can reproduce our work according to the methodology introduced in the paper;An example version of the CN-MSLU-100K dataset produced in this study, CN-MSLU-DEMO, has also been made available.A detailed description of all the data is presented below.<b>2. Content</b>1. The <b>figure </b>folder contains the original images from the paper.2. The <b>DCAI-CLUD</b> folder contains data and code related to the proposed framework. Contains four subfolders: (1) <b><i>“</i></b><b>Sample filtering based on size and location of parcels</b><b><i>”</i></b> corresponds to chapters 2.1 and 3.2 of the paper. (2) <b>BDF-Net Lite</b> is a simplified version of <b>BDF-Net</b>, the land use classification model used for the proposed human-computer collaborative data annotation in this study.The BDF-Net Lite folder includes the code used for model training and prediction as well as a demo dataset. (3) “<b>Statistics and analysis of the dataset construction process</b>” corresponds to chapters 2.4 and 3.4 of the paper.<b> </b> (4) <b>CN-MSLU-DEMO</b> is a demo of <b>CN-MSLU-100K,</b> which is a dataset of 100,000 land-use samples constructed in this study. The CN-MSLU-DEMO folder includes both a demo dataset and the starter code to explore the data.<b>3. </b><b>Sample filtering based on size and location of parcels</b>The folder contains two csv files:"<b>sample by area.csv</b>" is used to count the parcels by area, each line represents a parcel. There are 4 fields in the table: id, area, label and in_urban. When the "in_urban" field is 1, it means the current parcel is in the urban area, and when it is 0, it means the parcel is not in the urban area. The reader can group the parcels based on the "area" field and then calculate the metrics in the paper based on the “label”and the "in_urban" field."<b>filtering by location.csv</b>" is the statistics of the "filtering by location" method proposed in the paper according to the dispersion factor, each row represents a parcel. There are 4 fields in the table: id, Dispersion_factor, label and in_urban. The "dispersion_factor" field is incremented from 0~1 in 0.1 steps. The reader can group parcels based on the “dispersion_factor” field and then calculate the metrics in the paper based on the “label” and “in_urban” fields.<b>4. BDF-Net Lite</b><b>3.1 Overview</b>The classification model used in DCAI-CLUD was <b>BDF-Net</b>, a two-branch neural network coupling remote sensing images and POI data. But BDF-Net is just an example model here because any classification model can be used for pre-labeling the data.However, <b>BDF-Net Lite</b> here is just a simplified version of BDF-Net, as the full version of BDF-Net used in the paper is from another of our unpublished articles. Nonetheless, this still helps the reader understand our work because DCAI-CLUD is entirely independent of the models embedded in it.<b>3.2 Content</b>The "<b>data</b>" folder contains the example data. Among them, the "tif" folder is the remote sensing image. "dataset.csv" is the dataset for model training, including the storage location of remote sensing images and poi embedding results of the samples.The "<b>model</b>" folder is the model structure defined using Pytorch.The "<b>run</b>" folder is used to store the output file of each training result of the model, and its sub-folders are named with the name of the model file used for each training, such as "BDF_Net_Lite." Subfolders of the "BDF_Net_Lite" folder are named after the time of each training session, such as "20240110_2123". The folder "20240110_2123" contains the folder "weights" and the folder "writers," "model_this_time.py," and "current.yaml." The "weights" folder stores the model weights. The "writers" folder stores the tensorboard file that records the training parameters of the model. "model_this_time.py" is the model file used for this training. "current. yaml" records the parameter settings for this training.The "<b>Tools</b>" folder is the toolkit required for modeling training.The "<b>yamls</b>" folder stores the parameter settings for model training. Among them, the "Training" folder is the parameter settings for each model training, and "yaml_parent.yaml" is the base parameter that will be referenced automatically.The "<b>Run.py</b>" is the code that executes the model training. It is also the entry point of the main program. The method of calling the program has been noted in the code.The “<b>requirements.txt</b>” file is the dependencies that are needed to run the code. Install the runtime environment with the command "pip install -r requirements.txt".The environment required for this project includes:Python 3.7; Pytorch; Pandas; Numpy; Matplotlib, etc.<b>5.</b><b>Statistics and analysis of the dataset construction process</b>There are two subfolders in the folder:<b>Misleading effect of pre-labeling</b>: This folder contains an Excel table "<b>Results of the experiment.xlsx</b>", which is the statistical data of the experiment about the misleading effect of pre-labeling in the paper. Each row of data in the table represents a labeling record containing the model's pre-labeling labels, the labels chosen by the labeler when pre-labeling was available, the labels chosen by the labeler when no pre-labeling was available, and the true labels. The reader can reproduce our results based on this data and the methods described in the paper.<b>Pearson correlation of labeler indicators</b>: Here are the data and code of the experiments on the statistics of labeler indicators. For detailed instructions, please refer to the "README.md" file inside.<b>6. CN-MSLU-DEMO folder</b><b>2.1 Overview</b>The <b>CN-MSLU-100K</b> dataset consists of over 100,000 irregular remote sensing land parcel images. Combining the "Classification and Planning Standards for Urban Land Use" (GB 50137-2011)(https://sthjj.jiujiang.gov.cn/zwgk_215/zcjd/202006/P020200623535114893887.pdf) and Alibaba's "AMAP POI Classification Standard"(https://lbs.amap.com/api/ios-sdk/guide/map-data/poi/), we have categorized the main features depicted in the remote sensing images into 5 major classes and 22 sub-classes. Additionally, we have included the "Transportation" category, which has fewer labeled instances, as well as the "Unknown Land Use" category, which is challenging to determine due to insufficient plot information. In total, there are 7 major classes and 28 sub-classes.<b>2.2 Content</b>The folder " <b>Classification</b>" contains all the data description files in XML format, including `category`, `path`, `image size`, and other information.The folder " <b>ImageSets</b>" stores the original images of the dataset, organized in different directory by category. We extracted 3 images from each of the five main categories to produce the CN-MSLU-DEMO dataset.The python file “<b>DatasetGenerate.py</b>” is an demo code that organizes all the XML files into csv format dataset. This python script needs to be run in a Python 3.x environment.The csv file “<b>CN-MSLU-DEMO.csv</b>”is the dataset generated by “DatasetGenerate.py”. The code reads XML files using multiple threads and saves the extracted data as a CSV file for dataset creation.The markdown file “README-en.md” is a detailed description of this dataset and the code.Due to the terms and conditions of the map service not allowing us to publish map information, we sampled the data and removed their geographic information. However, this demo dataset still allows interested researchers to better understand the characteristics and applicability of the dataset.