Issue 1:
Add a command-line flag to optionally ignore headers passed in subsequent CSV-import files.

As per discussion in Slack, it would be very helpful to allow CSV headers in every neo4j-admin import file. For example, in the (truncated) command:

./bin/neo4j-admin import --id-type STRING --nodes:Nodes "mynodes-\d+.csv" ...

If I have only one file named mynodes-0.csv, then all works as planned. But if I have mynodes-0.csv and mynodes-1.csv, then mynodes-1.csv must not have headers. This seems a bit silly, since it's easy enough to ignore — and it can be a huge hassle to drop that line from every incoming CSV.

As an example user-story: Dask's to_csv function, for example, is a prime candidate source for Neo4j data-import; for data consistency, this API adds a header to all (or, optionally, none) of the output CSVs.

Potential Short Term Fix

If you found this because you're running into this problem: You can simply remove headers from ALL files, and then pass ANOTHER CSV with just headers; but, of course, this is pretty frustrating :)

And one more thing:

And one more thing! The documentation currently states:

Multiple files will be logically seen as one big file from the perspective of the importer. The first line must contain the header. Multiple data sources like these can be specified in one import, where each data source has its own header. [Emphasis mine.]

This "first line must contain the header" and "each data source has its own header" is, I think, misleading: Each "data source" isn't clearly different (at least to a beginner) than "data file". That suggests that each file may have its own header, but that's not the case!

An end to our sorrows?

Perhaps a --ignore-firstline flag (or something more elegant) could be passed to the importer to indicate that every file has a header? It's probably even possible to do this automatically, since the first line of every file will be the same. That should be easy to detect! But I appreciate that there might be some corner-case where existing users rely on that, and would suffer if the native behavior were changed.

Issue 2:
Feature request

Add an option in neo4j-import to skip the header line in very large CSV files, for use when separate header files are used.

Scenario

In enterprise data work, it's not unusual to have a data pipeline where one team is doing data extraction and shipping very large CSVs to another team tasked with building the Neo4j graph. Neo4j-import lets the graph team design compact header files so that the very large CSVs do not have to be opened and edited (which is great!). Currently, neo4j-import requires that no header line is present in the very large CSVs to be loaded using separate header files.

The problem is that it's difficult to diagnose extract issues on deep and wide files missing headers when imports fail. So for very large files, it's a better practice to keep the headers on so that they can be easily validated against the designed header file CSVs that carry the key assignments, data types etc. Under the current set of options, either the large CSV has to be generated without a header line (adding IT mystery) or the graph team has to go in and remove the header line once the extract is validated (adding Dev hassle).

Currently, if the large CSV has a header row, neo4j-import will attempt to create a node using the header data as property values - which may not always work if conflicting data types are set in the header file.

Possible implementation

C:/neo4j/bin/neo4j-import --skip-header-line TRUE \
--into graph.db \
--nodes:MyLabel: "My_Header.csv, My_Very_Large_File.csv"

default value would be --skip-header-line FALSE to be consistent with basic usage without header CSV files.

Expected behavior

CSV import would start at first data line of My_Very_Large_File.csv, skipping the header line.

The header values are supplied by My_Header.csv, as per usual.

Note

A more general case would be:

--start-import-at-line n

where n is any arbitrary line number in the file...

Thanks,

Michael