Repository: frappe/frappe

Issue:
Data Import trouble with different child tables having different lengths
The illustration of this issue uses Purchase Invoices in ERPNext, but as it has to do with the basics of Data Import, I believe it is a Frappe-level problem.

I am in process of migrating an existing Xero ledger to ERPNext for a small not-for-profit. We have converted Xero's export of all "Bills" to match the import template for Purchase Invoices. However, the import fails, complaining about various fields being mandatory. I attach a sample of four Purchase Invoice records and the resulting output to illustrate the problem:
[Test_Import_Purchase_Invoice_csv.txt](https://github.com/frappe/frappe/files/5046109/Test_Import_Purchase_Invoice_csv.txt)
![Screenshot_2020-08-08_12-12-56](https://user-images.githubusercontent.com/3825429/89718016-c43b3300-d970-11ea-972a-04dcb8585f56.png)
![Screenshot_2020-08-08_12-13-50](https://user-images.githubusercontent.com/3825429/89718015-c43b3300-d970-11ea-937a-620af97ef900.png)

In more detail:
This is on a current (just did bench update today, to be sure) develop installation of Frappe/ERPNext. I have a custom DocType, named Fund, to represent not-for-profit accounting Funds, and have defined it as an Accounting Dimension. I am trying to import (with the Data Import tool) the Purchase Invoices from Xero, and I provide a sample of four such conversions in the attached CSV file.

Note that the entire file previews fine when I upload it in the Data Import tool, as shown in the first screenshot. But when I click on start import, it generates several warning/error messages, as shown in the second screenshot, and no records are imported.

Looking at the details, you can see that the first Purchase Invoice generates no messages. It happens to have the same number of child Items as it has child Purchase Taxes and Charges.

The second Purchase Invoice generates a message that Account Head and Description are mandatory for Purchase Taxes and Charges, but only for its third line (row 6) where there is an Item but no Purchase Tax and Charge (because there happen to be three Items and only two Purchase Taxes and Charges).

The third Purchase Invoice shows that a record with no Purchase Taxes and Charges at all generates the same problem, even if it has only one Item.  Of course, not every invoice has any Purchase Taxes and Charges, so there needs to be some way to import such records.

And finally, the fourth Purchase Invoice has two Purchase Taxes and Charges, but only one Item, and in this case Frappe complains about multiple Item fields being mandatory -- but only for the line (row 9) where there is a Purchase Taxes and Charges, but no Item.

Together, these examples produce a picture that Frappe simply cannot handle the case in which two different child tables of a record being imported are different lengths.

As this is a show stopper for a rather urgent data import for our organization, I will begin work on a fix immediately, and of course will submit a pull request if I get a reasonable fix working. But any input, pointers, or possible fixes are very welcome in the meantime! Thanks. 

Data Import trouble with different child tables having different lengths
The illustration of this issue uses Purchase Invoices in ERPNext, but as it has to do with the basics of Data Import, I believe it is a Frappe-level problem.

I am in process of migrating an existing Xero ledger to ERPNext for a small not-for-profit. We have converted Xero's export of all "Bills" to match the import template for Purchase Invoices. However, the import fails, complaining about various fields being mandatory. I attach a sample of four Purchase Invoice records and the resulting output to illustrate the problem:
[Test_Import_Purchase_Invoice_csv.txt](https://github.com/frappe/frappe/files/5046109/Test_Import_Purchase_Invoice_csv.txt)
![Screenshot_2020-08-08_12-12-56](https://user-images.githubusercontent.com/3825429/89718016-c43b3300-d970-11ea-972a-04dcb8585f56.png)
![Screenshot_2020-08-08_12-13-50](https://user-images.githubusercontent.com/3825429/89718015-c43b3300-d970-11ea-937a-620af97ef900.png)

In more detail:
This is on a current (just did bench update today, to be sure) develop installation of Frappe/ERPNext. I have a custom DocType, named Fund, to represent not-for-profit accounting Funds, and have defined it as an Accounting Dimension. I am trying to import (with the Data Import tool) the Purchase Invoices from Xero, and I provide a sample of four such conversions in the attached CSV file.

Note that the entire file previews fine when I upload it in the Data Import tool, as shown in the first screenshot. But when I click on start import, it generates several warning/error messages, as shown in the second screenshot, and no records are imported.

Looking at the details, you can see that the first Purchase Invoice generates no messages. It happens to have the same number of child Items as it has child Purchase Taxes and Charges.

The second Purchase Invoice generates a message that Account Head and Description are mandatory for Purchase Taxes and Charges, but only for its third line (row 6) where there is an Item but no Purchase Tax and Charge (because there happen to be three Items and only two Purchase Taxes and Charges).

The third Purchase Invoice shows that a record with no Purchase Taxes and Charges at all generates the same problem, even if it has only one Item.  Of course, not every invoice has any Purchase Taxes and Charges, so there needs to be some way to import such records.

And finally, the fourth Purchase Invoice has two Purchase Taxes and Charges, but only one Item, and in this case Frappe complains about multiple Item fields being mandatory -- but only for the line (row 9) where there is a Purchase Taxes and Charges, but no Item.

Together, these examples produce a picture that Frappe simply cannot handle the case in which two different child tables of a record being imported are different lengths.

As this is a show stopper for a rather urgent data import for our organization, I will begin work on a fix immediately, and of course will submit a pull request if I get a reasonable fix working. But any input, pointers, or possible fixes are very welcome in the meantime! Thanks. 



Discussion:

Resolving patch:
diff --git a/frappe/core/doctype/data_import/importer.py b/frappe/core/doctype/data_import/importer.py
--- a/frappe/core/doctype/data_import/importer.py
+++ b/frappe/core/doctype/data_import/importer.py
@@ -465,6 +465,8 @@ def parse_next_row_for_import(self, data):
 
 				if doctype != self.doctype and table_df:
 					child_doc = row.parse_doc(doctype, parent_doc, table_df)
+					if child_doc is None:
+						continue
 					parent_doc[table_df.fieldname] = parent_doc.get(table_df.fieldname, [])
 					parent_doc[table_df.fieldname].append(child_doc)
 
@@ -570,6 +572,11 @@ def __init__(self, index, row, doctype, header, import_type):
 	def parse_doc(self, doctype, parent_doc=None, table_df=None):
 		col_indexes = self.header.get_column_indexes(doctype, table_df)
 		values = self.get_values(col_indexes)
+
+		if all(v in INVALID_VALUES for v in values):
+			# if all values are invalid, no need to parse it
+			return None
+
 		columns = self.header.get_columns(col_indexes)
 		doc = self._parse_doc(doctype, columns, values, parent_doc, table_df)
 		return doc


Test patch: