Package: dgraphpandas
dgraphpandasA Library (with accompanying cli tool) to transformPandasDataFrames into Exports (RDF) to be sent toDGraph Live LoaderdgraphpandasUsageCommand LineModuleGetting StartedHorizontal and Vertical FormatsHorizontalVerticalEdgesConfigurationAdditional ConfigurationSchema and TypesGenerating a SchemaGenerating TypesSamplesWorking with Larger FilesCommand LineModuleLocal SetupUsagepython-mpipinstalldgraphpandasCommand Line❯dgraphpandas--help
usage:dgraphpandas[-h][-x{upserts,schema,types}][-fFILE]-cCONFIG[-ckCONFIG_FILE_KEY][-oOUTPUT_DIR][--console][--export_csv][--encodingENCODING][--chunk_sizeCHUNK_SIZE][--gz_compression_levelGZ_COMPRESSION_LEVEL][--key_separatorKEY_SEPARATOR][--add_dgraph_type_recordsADD_DGRAPH_TYPE_RECORDS][--drop_na_intrinsic_objectsDROP_NA_INTRINSIC_OBJECTS][--drop_na_edge_objectsDROP_NA_EDGE_OBJECTS][--illegal_charactersILLEGAL_CHARACTERS][--illegal_characters_intrinsic_objectILLEGAL_CHARACTERS_INTRINSIC_OBJECT][--version][-v{DEBUG,INFO,WARNING,ERROR,NOTSET}]This is a real example which you can find in thesamples folderand run from the root of this repository:dgraphpandas\--configsamples/planets/dgraphpandas.json\--config_file_keyplanet\--filesamples/planets/solar_system.csv\--outputsamples/planets/outputModuleThis example can also be found inNotebookform.importdgraphpandasasdpd# Define a Configuration for your data files(s). Explained further in the Configuration section.config={"transform":"horizontal","files":{"planet":{"subject_fields":["id"],"edge_fields":["type"],"type_overrides":{"order_from_sun":"int32","diameter_earth_relative":"float32","diameter_km":"float32","mass_earth_relative":"float32","mean_distance_from_sun_au":"float32","orbital_period_years":"float32","orbital_eccentricity":"float32","mean_orbital_velocity_km_sec":"float32","rotation_period_days":"float32","inclination_axis_degrees":"float32","mean_temperature_surface_c":"float32","gravity_equator_earth_relative":"float32","escape_velocity_km_sec":"float32","mean_density":"float32","number_moons":"int32","rings":"bool"},"ignore_fields":["image","parent"]}}}# Perform a Horizontal Transform on the passed file using the config/key# Generate RDF Upsert statementsintrinsic,edges=dpd.to_rdf('solar_system.csv',config,'planet',output_dir='.',export_rdf=True)# Do something with these statements e.g write to zip and ship to DGraph# The cli will zip this output automatically# In module mode when you provide output_dir and export_rdf it will automatically zip and write to diskprint(intrinsic)print(edges)Alternatively, you could call the underlying methods# Perform a Horizontal Transform on the passed file using the config/keyintrinsic,edges=horizontal_transform('solar_system.csv',config,"planet")# Generate RDF Upsert statementsintrinsic_upserts,edges_upserts=generate_upserts(intrinsic,edges)Getting StartedFire up a Terminal (Assuming not Windows but WSL is okay).# Create a Configurationecho'{"transform": "horizontal","files": {"animal": {"subject_fields": ["species_id"],"edge_fields": ["habitat_id"],"type_overrides": {"number_of_legs": "int32"}}}}'>dgraphpandas.json# Create a Data fileecho"species_id,name,number_of_legs,habitat_id,1,Elephant,4,10,2,Lion,4,7,3,Flamingo,2,78,">animals.csv# Run dgraphpandasdgraphpandas-fanimals.csv-cdgraphpandas.json-ckanimal# Unzip outputgzip-danimals_intrinsic.gz
gzip-danimals_edges.gz# Verify Output❯catanimals_intrinsic
<animal_1><name>"Elephant"^^<xs:string>.
<animal_2><name>"Lion"^^<xs:string>.
<animal_3><name>"Flamingo"^^<xs:string>.
<animal_1><number_of_legs>"4"^^<xs:int>.
<animal_2><number_of_legs>"4"^^<xs:int>.
<animal_3><number_of_legs>"2"^^<xs:int>.
<animal_1><dgraph.type>"animal"^^<xs:string>.
<animal_2><dgraph.type>"animal"^^<xs:string>.
<animal_3><dgraph.type>"animal"^^<xs:string>.

❯catanimals_edges
<animal_1><habitat><habitat_10>.
<animal_2><habitat><habitat_7>.
<animal_3><habitat><habitat_78>.Horizontal and Vertical Formatsdgraphpandas takes two kinds of input; vertical and horizontal. In both instances they are expected to be in csv format.HorizontalHorizontal follows a tabular structure and is probably the more likely format found out in the wild. It might look like this:customer_id    weight    height
1              90        190
2              23        120
3              100       56When you provide the subject fields as['customer_id']then dgraphpandas will be treat the rest of the columns as data values. It will be pivoted like this:customer_id    predicate    object
1              weight       90
1              height       190
2              weight       23
2              height       120
3              weight       100
3              height       56Then along with the options provided within the passed configuration then the output RDF might look like this:<customer_1><weight>"90"^^<xs:int>.<customer_1><height>"190"^^<xs:int>.<customer_2><weight>"23"^^<xs:int>.<customer_2><height>"120"^^<xs:int>.<customer_3><weight>"100"^^<xs:int>.<customer_3><height>"56"^^<xs:int>.Wherecustomer_was appended as it was defined as thetypefor this export and types were associated because it was defined insidetype_overrides.VerticalVertical transformation is very similar to the above Horizontal explanation but we skip the initial pivoting step as the data is already looks likecustomer_id,predicate,object.EdgesEdges are derived from theedge_fieldsdefined inside the file level configuration and they are sent just like data fields from the input file.As they are defined inedge_fields, dgraphpandas will split these out and treat them slightly differently during transformation and generation of the RDF output.For example if we had an E-Commerce Orders table:order_id    customer_id    store_id
5           1              1
9           2              2
2           3              1And we had a configuration like this:{"transform":"horizontal","files":{"order":{"subject_fields":["order_id"],"edge_fields":["customer_id","store_id"]}}}Then the output RDF would look like this:<order_5><customer><customer_1>.<order_9><customer><customer_2>.<order_2><customer><customer_3>.<order_5><store><store_1>.<order_9><store><store_2>.<order_2><store><store_1>.Where each of the orders has been associated with it's customer and store.ConfigurationA Configuration file influences how we transform a DataFrame. It consists of:Global configuration optionsOptions which will be applied to all filesThese can either be defined in the configuration or askwargsin the transform method or both where thekwargstakes priority.A collection offilesFile configuration optionsOptions which will be applied only to this entrysubject_fieldsis required so the unique identifier for a row in the DataFrame can be foundedge_fieldsare optional and if provided will generate edge outputtype_overridesare optional but recommended to ensure the correct type is attached in RDFIf you are running this with the module and passing viakwargsthen the above options may also be lambda callable that takes the DataFrame as an input. For example if you didn't want to hard code all your edge fields and were following a convention that all edge fields have suffix_idthen you could set the edge_fields tolambda frame: frame.loc[frame['predicate'].str.endswith('_id'), 'predicate'].unique().tolist(). For this specific convention, it's common enough to have it's own built in option. Seeedge_id_conventionAn example of the configuration for theplanets samplemight look like this:config={"transform":"horizontal","files":{"planet":{"subject_fields":["id"],"edge_fields":["type"],"type_overrides":{"order_from_sun":"int32","diameter_earth_relative":"float32","diameter_km":"float32","mass_earth_relative":"float32","mean_distance_from_sun_au":"float32","orbital_period_years":"float32","orbital_eccentricity":"float32","mean_orbital_velocity_km_sec":"float32","rotation_period_days":"float32","inclination_axis_degrees":"float32","mean_temperature_surface_c":"float32","gravity_equator_earth_relative":"float32","escape_velocity_km_sec":"float32","mean_density":"float32","number_moons":"int32","rings":"bool"},"ignore_fields":["image","parent"]}}}Additional ConfigurationGlobal LevelThese options can be placed on the root of the config or passed askwargsdirectly.add_dgraph_type_recordsDGraph has a special predicate calleddgraph.type, this can be used to query via thetype()function. Ifadd_dgraph_type_recordsis enabled, then we adddgraph.typefields
to the current export.strip_id_from_edge_namesIts common for a data set to have a reference to another 'table' using_idconventionYou may not want the_idin your predicate name so this options strips it awayFor example if you had a Student & School then the student might more sense to have(Student) - school -> (School)rather then(Student) - school_id -> (School).drop_na_intrinsic_objectsAutomatically drop intrinsic records where the object is NA. In a relational model, you might have a column with anullentry however in a graph model you may want to omit the attribute altogether.drop_na_edge_objectsSame asdrop_na_intrinsic_objectsbut for edges.key_separatorSeparator used to combine key fields. For example if the key separator was_and we were operating on an intrinsic attribute for a customer with id 1 then thexidwould becustomer_1but if our seperator was$then it would becustomer$1.illegal_charactersCharacters to strip from intrinsic and edge subjects. if the unique identifier has a character not supported by RDF/DGraph then strip them away or they will not be accepted by live loading.illegal_characters_intrinsic_objectSame asillegal_charactersbut for the subject on intrinsic fields. These have a different set of illegal characters because subjects on intrinsic records are actual data values and are quoted. They therefore can accept many more characters then the subject.ensure_xid_predicateSchema generation option to ensure that thexidpredicate is applied to the schema. If you use the--upsertPredicate xidthen this must be set so that the predicate is created and indexed.File Leveltype_overridesRecommended. This ensures that data types are being treated as a type and the output RDF has the correct type mapped into it. Without this fields will go under the default rdf type<xs:string>but you may want a field to be a true int in RDF.Additionally certain data types such asdatetime64will activate special handling to ensure the output in RDF is within the correct format to be ingested into DGraph.Supported Types can be foundherecsv_edgesSometimes a vendor will provide a data file where a single column is actually a csv list and each csv value should be broken into multiple RDF statements (because they relate to independent entities). Adding that column into this list will do that.For example in thenetflix sample / title filewe have acastcolumn where the values areactor_1, actor2. Enablingcsv_edgeswill ensure that the movie has 2 different relationships for each cast member.csv_edges_separatorAlternative separator forcsv_edgesignore_fieldsAdd fields in the input that we don't care about to this list so they aren't present in the outputoverride_edge_nameEnsure that the edge name as a different predicate and/or target_node_type to what is defined in the file.For example in thepokemon sample / pokemon_speciesfile you will see a column calledevolves_from_specieswhich tells us for a given pokemon which other pokemon does it evolve from. If we were to use the raw data here we would get aevolves_from_speciesedge with an incorrect target xid. Instead we want to override thetarget_node_typetopokemonso the edge correctly loops back to a node of the same type.pre_renameRename intrinsic predicates or edge names to something elseread_csv_optionsApplied to thepd.read_csvcall when a file is passed to a transformFor example if the vendor file was tab separated then this could be{'sep': '\t'}date_fieldsApply datetime options to a field. This option can be useful when the input file has a date column with an unsual format. For each field, this object is passed intopd.to_datetime. For example if you had a column calleddobthen you could set this object to{ "dob": {"format": "%Y-%m-%d"} }. All thestandardformat codes are supported.edge_id_conventionApplies_idconvention to find edges when set totrueSame as providing the edge_fieldlambda frame: frame.loc[frame['predicate'].str.endswith('_id'), 'predicate'].unique().tolist().predicate_fieldOnly applicable for vertical transformsAllows you to define your own predicate field name if not the defaultpredicateobject_fieldOnly applicable to vertical transformsAllows you to define your own object field name if not the defaultobjectoptionsAdditional Options for Schema generation such as indexes or other directives.This is a key value pair between a intrinsic/edge to list of directives to applye.g"title": ["@index(exact, fulltext)", "@count"]list_edgesSchema option to define an edge as a list. This will ensure the type is[uid]rather then justuidSchema and TypesGenerating a SchemaDGraph allows you to define aschema. This can be generated using the same configuration used above but there are also additional options you can add such asoptionsandlist_edgeswhich are exclusively used for schema generation.# Model the data, define types, edges and any options>echo'{"transform": "horizontal","files": {"animal": {"subject_fields": ["species_id"],"type_overrides": {"name": "string","legs": "int","weight": "float","height": "float","discovered": "datetime64","aquatic": "bool"},"edge_fields": ["class_id", "found_in"],"options": {"name": ["@index(hash)"],"discovered": ["@index(year)"],"class": ["@reverse"],"found_in": ["@reverse", "@count"]},"list_edges": ["found_in"]}}}'>dgraphpandas.json# Apply the config to the schema generation logic>dgraphpandas-cdgraphpandas.json-xschema# Inspect Schema>catschema.txt

found_in:[uid]@reverse@count.
aquatic:bool.
discovered:dateTime@index(year).
weight:float.
height:float.
legs:int.
name:string@index(hash).
species:string.
class:uid@reverse.# Apply to DGraphdgraphlive-sschema.txtGenerating TypesDGraph also allows you to definetypesthat can be used to categorize nodes. This can also be generated from the same configuration as data loading.# Model the data, define types, edges and any optionsecho'{"transform": "horizontal","files": {"animal": {"subject_fields": ["species_id"],"type_overrides": {"name": "string","legs": "int","weight": "float","height": "float","discovered": "datetime64","aquatic": "bool"},"edge_fields": ["class_id", "found_in"],"class": ["@reverse"],"found_in": ["@reverse", "@count"],"list_edges": ["found_in"]},"habitat": {"subject_fields": ["id"],"type_overrides": {"name": "string"}}}}'>dgraphpandas.json# Apply the config to the schema generation logic>dgraphpandas-cdgraphpandas.json-xtypes-vDEBUG# Inspect Types❯cattypes.txttypeanimal{found_in
aquatic
discovered
height
weight
legs
species
name
class}typehabitat{id
name}# Apply to DGraph# NOTE: you should always apply the schema# before applying types else dgraph# won't know what the predicates aredgraphlive-stypes.txtSamplesSamples can be foundhere. They follow a convention where the download script can be found within theinputdirectory and the config, generate_upsert, publish scripts can be found root of each respective sample.There are alsoJupyter Notebookswhich should show step by step examples.Working with Larger FilesIf you have very large input files, it may make sense to break up your files into smaller ones to reduce the likely hood of memory issues.dgraphpandas provides facilities to break up exports via the cli tool into chunks or if you are using the module directly then you can find an example below on how to use pandas to break up your file.Command LineIn the CLI you have thechunk_sizeparameter to determine an upper limit for your files.python-mdgraphpandas\-csamples/netflix/dgraphpandas.json\-cktitle-fsamples/netflix/input/netflix_titles.csv\-osamples/netflix/output\--chunk_size1000When you pass this, onlychunk_sizelines will be pushed through the RDF generation logic at a time and the output will be indexed per chunk. For example:❯ls-lasamples/netflix/output/
total12M
drwxr-xr-x2kirankiran4.0KApr418:13.
drwxr-xr-x6kirankiran4.0KApr416:45..
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges.gz
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges_2.gz
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges_3.gz
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges_4.gz
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges_5.gz
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges_6.gz
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges_7.gz
-rw-r--r--1kirankiran706KApr418:13netflix_titles_edges_8.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic_2.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic_3.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic_4.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic_5.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic_6.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic_7.gz
-rw-r--r--1kirankiran701KApr418:13netflix_titles_intrinsic_8.gzYou can then take these exports and live load them as normal.ModuleThechunk_sizemethod is also available onto_rdf. If you provide anoutput_dir&export_rdfthis will automatically be written out to an export file on disk.For Example:importdgraphpandasasdpddpd.to_rdf('your_input.csv',config,'your_input_key',output_dir='.',export_rdf=True,chunk_size=1000)If you wanted more control, then you could also call the underlying methods to leverage the fact that the transform methods can take aDataFramedirectly and you can pre-chunk before you enter.fromdgraphpandas.strategies.horizontalimporthorizontal_transformfromdgraphpandas.writers.upsertsimportgenerate_upserts# Each Chunk won't be loaded into memory until it hits that particular loop.forindex,frameinenumerate(pd.read_csv('your_input.csv',chunksize=1000)):# Generate for this Chunkintrinsic,edges=horizontal_transform(frame,dgraphpandas_config,'your_input_key')# Generate Rdf Upserts for this Chunkintrinsic_upserts,edges_upserts=generate_upserts(intrinsic,edges)# Then you can do whatever you want with these before the next iterationLocal SetupAssuming you have already cloned the repo and have a terminal in the root of the project.# Create Virtual Environment and Activate itcondacreate-ndgraphpandaspython=3.6# or venvcondaactivatedgraphpandas# Restore packagespython-mpipinstall-rrequirements-dev.txt
python-mpipinstall-rrequirements.txt# Run Flakeflake8--count.# Run Testspython-munittest# Create & Run DGraphdocker-composeup# Try a Sample# See Sample section for more details# It should help getting some data,# generating rdf and publishing to your# local DGraph# If you are making changes then# Install a Local Copy of the Librarypython-mpipinstall-e.# Remember to uninstall once donepython-mpipuninstalldgraphpandas-y