Project: GraphScope (App Definition and Development / Database)
File: data_import.md

In our guide on , we introduced the basics of importing graph data using a simple YAML configuration. This section delves deeper, providing a thorough exploration of the extensive configuration options available for data import. Currently we only support import data to graph from local `csv` files or `odps` table. See configuration `loadingconfig.datasource.scheme`. To illustrate, let's examine the `examples/modernimportfull.yaml` file. This configuration is designed for importing the "modern" graph and showcases the full range of configuration possibilities. We'll dissect each configuration item in the sections that follow. ``` yaml loading_config: data_source: scheme: file location: /home/modern_graph/ format: metadata: delimiter: "|" header_row: true quoting: false quote_char: '"' double_quote: true escape_char: '\' block_size: 4MB vertex_mappings: type_name: person inputs: person.csv column_mappings: column: index: 0 name: id property: id column: index: 1 name: name property: name column: index: 2 name: age property: age type_name: software inputs: software.csv column_mappings: column: index: 0 name: id property: id column: index: 1 name: name property: name column: index: 2 name: lang property: lang edge_mappings: type_triplet: edge: knows source_vertex: person destination_vertex: person inputs: personknowsperson.csv sourcevertexmappings: column: index: 0 name: id destinationvertexmappings: column: index: 1 name: id column_mappings: column: index: 2 name: weight property: weight type_triplet: edge: created source_vertex: person destination_vertex: software inputs: personcreatedsoftware.csv sourcevertexmappings: column: index: 0 name: id destinationvertexmappings: column: index: 1 name: id column_mappings: column: index: 2 name: weight property: weight ``` ```yaml graph: dd_graph loading_config: data_source: scheme: odps # file, odps import_option: init # init, overwrite format: type: arrow vertex_mappings: type_name: person # must align with the schema inputs: yourprojname/tablename/partitioncolname=partitionname column_mappings: column: index: 0 name: id property: id column: index: 1 name: name property: name column: index: 2 name: age property: age type_name: software inputs: yourprojname/tablename/partitioncolname=partitionname column_mappings: column: index: 0 name: id property: id column: index: 1 name: name property: name column: index: 2 name: lang property: lang edge_mappings: type_triplet: edge: knows source_vertex: person destination_vertex: person inputs: yourprojname/tablename/partitioncolname=partitionname sourcevertexmappings: column: index: 0 name: srcuserid property: id destinationvertexmappings: column: index: 1 name: dstuserid property: id column_mappings: column: index: 2 name: weight property: weight type_triplet: edge: created source_vertex: person destination_vertex: software inputs: yourprojname/tablename/partitioncolname=partitionname sourcevertexmappings: column: index: 0 name: id destinationvertexmappings: column: index: 1 name: id column_mappings: column: index: 2 name: weight property: weight ``` The table below offers a detailed breakdown of each configuration item. In this table, we use the "." notation to represent the hierarchy within the YAML structure. | Key | Default | Descriptions | Mandatory | | -- | -- | -- |-- | | loading_config | N/A | Loading configurations | Yes | | loadingconfig.datasource | N/A | Place that maintains the raw data | Yes | | loadingconfig.datasource.location | N/A | Path to the data source in the container, which must be mapped from the host machine while initializing the service | Yes | loading_config.scheme | file | The source of input data. Currently only `file` and `odps` are supported | No | | loading_config.format | N/A | The format of the raw data in CSV | Yes | | loading_config.format.metadata | N/A | Mainly for configuring the options for reading CSV | Yes | |

| '\|' | Delimiter used to split a row of data | Yes | | loadingconfig.format.metadata.headerrow | true | Indicate if the first row should be used as the header | No | | loading_config.format.metadata.quoting | false | Whether quoting is used | No | | loadingconfig.format.metadata.quotechar | '\"' | Quoting character (if `quoting` is true) | No | | loadingconfig.format.metadata.doublequote | true | Whether a quote inside a value is double-quoted | No | | loading_config.format.metadata.escaping | false | Whether escaping is used | No | | loadingconfig.format.metadata.escapechar | '\\' | Escaping character (if `escaping` is true) | No | | loadingconfig.format.metadata.batchsize | 4MB | The size of batch for reading from files | No | | | | | | | vertex_mappings | N/A | Define how to map the raw data into a graph vertex in the schema | Yes | | vertexmappings.typename | N/A | Name of the vertex type | Yes | | vertexmappings.inputs | N/A | List of files containing raw data for the vertex type, relative to `loadingconfig.data_source.location` | Yes | | vertexmappings.columnmappings | The columns in the raw data will be automatically mapped to the properties, in given order | Define which column of the vertex data maps to a given property | No | | vertexmappings.columnmappings.column.index | N/A | Index of the column in the raw data | No | | vertexmappings.columnmappings.column.name | N/A | Name of the column in the raw data | No | | vertexmappings.columnmappings.property | N/A | Property in the schema to which the column maps. | No | | | | | | | edge_mappings | N/A | Define how to map the raw data into a graph edge in the schema | Yes | | edgemappings.typetriplet | N/A | A triplet that uniquely defines an edge | Yes | | edgemappings.typetriplet.edge | N/A | The type of the edge | Yes | | edgemappings.typetriplet.source_vertex | N/A | Type of the source vertex for the edge | Yes | | edgemappings.typetriplet.destination_vertex | N/A | Type of the destination vertex for the edge | Yes | | edgemappings.inputs | N/A | List of files containing raw data for the edge type, relative to `loadingconfig.data_source.location` | Yes | | edgemappings.sourcevertex_mappings | The first column of the edge data | Define which columns of the edge data map to its source vertex. | No | | edgemappings.sourcevertex_mappings.column.index | N/A | Index of the column in the raw data representing the source vertex. | No | | edgemappings.sourcevertex_mappings.column.name | N/A | Name of the column in the raw data representing the source vertex. | No | | edgemappings.destinationvertex_mappings | The second column of the edge data | Define which columns of the edge data map to its source vertex. | No | | edgemappings.destinationvertex_mappings.column.index | N/A | Index of the column in the raw data representing the destination vertex. | No | | edgemappings.destinationvertex_mappings.column.name | N/A | Name of the column in the raw data representing the destination vertex. | No | | edgemappings.columnmappings | The columns in the raw data will be automatically mapped to the properties, in given order | Define which column of the edge data maps to a given property. | No | |

| N/A | Index of the column in the raw data | No | | edgemappings.columnmappings.column.name | N/A | Name of the column in the raw data | No | | edgemappings.columnmappings.property | N/A | Property in the schema to which the column maps. | No (but must present if column presents) | The loading_config section defines the primary settings for data loading. This includes specifying the data source, its format, and other metadata. loadingconfig.datasource.location: This is the path to the data source within the container. It's essential to ensure this location is mapped from the host machine when initializing the service. loading_config.format.metadata: Mainly for configuring the options for reading CSV loading_config.format.metadata.delimiter: Defines the delimiter used to split rows of data. loadingconfig.format.metadata.headerrow: Indicate if the first row should be used as the header. loading_config.format.metadata.quoting: Whether quoting is used loadingconfig.format.metadata.quotechar: Quoting character (if `quoting` is true) loadingconfig.format.metadata.doublequote: Whether a quote inside a value is double-quoted loadingconfig.format.metadata.escapechar: Whether escaping is used loadingconfig.format.metadata.escapechar: Escaping character (if `escaping` is true) loadingconfig.format.metadata.blocksize: The size of batch for reading from files The vertex_mappings section outlines how raw data maps to graph vertices based on the defined schema. vertexmappings.typename: Specifies the name of the vertex type. vertexmappings.inputs: Lists the files containing raw data for the vertex type. These paths are relative to `loadingconfig.data_source.location`. vertexmappings.columnmappings.column: Define which column of the vertex data maps to a given property. If not give, the columns in the raw data will be automatically mapped to the properties, in given order. vertexmappings.columnmappings.column.index: Index of the column in the raw data. vertexmappings.columnmappings.column.name: Name of the column in the raw data. vertexmappings.columnmappings.property: Property in the schema to which the column maps. The edge_mappings section details how raw data maps to graph edges as per the schema. edgemappings.typetriplet: A triplet that uniquely defines an edge. edgemappings.typetriplet.edge: Indicates the type of the edge. edgemappings.typetriplet.source_vertex: Indicates the type of the source vertex. edgemappings.typetriplet.destination_vertex: Indicates the type of the destination vertex. edgemappings.inputs: Lists the files containing raw data for the edge type. These paths are relative to `loadingconfig.data_source.location`. edgemappings.sourcevertex_mappings: Define which columns (can be multiple) of the edge map to its source vertex. If not given, the first column of the edge data will be used by default. edgemappings.sourcevertex_mappings.column.index: Index of the column in the raw data representing the source vertex. edgemappings.sourcevertex_mappings.column.name: Name of the column in the raw data representing the source vertex. edgemappings.destinationvertex_mappings: Define which columns (can be multiple) of the edge map to its destination vertex. If not given, the second column of the edge data will be used by default. edgemappings.destinationvertex_mappings.column.index: Index of the column in the raw data representing the destination vertex. edgemappings.destinationvertex_mappings.column.name: Name of the column in the raw data representing the destination vertex. edgemappings.columnmappings.column: Define which column of the edge data maps to a given property. If not give, the columns in the raw data will be automatically mapped to the properties, in given order. edgemappings.columnmappings.column.index: Index of the column in the raw data. edgemappings.columnmappings.column.name: Name of the column in the raw data. edgemappings.columnmappings.property: Property in the schema to which the column maps. By understanding and appropriately setting these configurations, users can seamlessly import their graph data into GraphScope, ensuring data integrity and optimal performance.