Repository: netbox-community/netbox

Issue:
Validate all headers/fields when bulk import objects
### NetBox version

v3.4.3

### Python version

3.8

### Steps to Reproduce

1. Open Prefix Bulk Import
2. Specify all required field options and add some dummy data
`prefix,status,testa,testb`
`10.10.10.0/24,active,a,b`
4. Press Import
5. Import is done without any complains


### Expected Behavior

Fields which are not stated in Field Options should throw an error, or a warning that some data was ignored.


### Observed Behavior

Import is done without any complains, Looks like all data was imported correct, although some fields were ignored


Discussion:
It this a regression? Did it used to throw an error? Are we sure this is actually wanted behavior before I look into this?
I just tried it in v3.09. It raises a `Unexpected column header "testa" found.` error.
Also checked versions 3.2.0, 3.3.0 and 3.4.0. The import works with version 3.4.0, before raises an `Unexpected column header "testa" found.` error.
@arthanson - Can you take a look at this as it's part of the bulk import stuff you changed.

I tried just adding the following validation on L147 here:

https://github.com/netbox-community/netbox/blob/56c7a238a4905d3220b63d8b33b07ca52df7efd1/netbox/utilities/forms/forms.py#L134-L153

```python
for header in headers.keys():
    if header not in self.fields:
        raise forms.ValidationError(f'Unexpected column header "{header}" found.')
```

But the ValidationError is caught here and discarded:

https://github.com/netbox-community/netbox/blob/56c7a238a4905d3220b63d8b33b07ca52df7efd1/netbox/netbox/views/generic/bulk_views.py#L461-L462
We should consider JSON/YAML imports as well as CSV. If we're concerned about checking for extraneous CSV headers (and IMO we should be), then this functionality should exist for JSON- and YAML-formatted data as well. Unlike CSV, which specifies a single row of headers, the fields for each object expressed in JSON or YAML can vary with each object, so each object will need to be validated individually. I think it's reasonable to replicate this pattern for CSV data, even if it's not technically necessary.

Resolving patch:
diff --git a/netbox/utilities/forms/bulk_import.py b/netbox/utilities/forms/bulk_import.py
--- a/netbox/utilities/forms/bulk_import.py
+++ b/netbox/utilities/forms/bulk_import.py
@@ -129,6 +129,7 @@ def _clean_csv(self, data, delimiter=CSVDelimiterChoices.AUTO):
         headers, records = parse_csv(reader)
 
         # Set CSV headers for reference by the model form
+        headers.pop('id', None)
         self._csv_headers = headers
 
         return records
diff --git a/netbox/utilities/forms/forms.py b/netbox/utilities/forms/forms.py
--- a/netbox/utilities/forms/forms.py
+++ b/netbox/utilities/forms/forms.py
@@ -70,22 +70,24 @@ class CSVModelForm(forms.ModelForm):
     """
     ModelForm used for the import of objects in CSV format.
     """
-    def __init__(self, *args, headers=None, fields=None, **kwargs):
-        headers = headers or {}
-        fields = fields or []
+    def __init__(self, *args, headers=None, **kwargs):
+        self.headers = headers or {}
         super().__init__(*args, **kwargs)
 
         # Modify the model form to accommodate any customized to_field_name properties
-        for field, to_field in headers.items():
+        for field, to_field in self.headers.items():
             if to_field is not None:
                 self.fields[field].to_field_name = to_field
 
-        # Omit any fields not specified (e.g. because the form is being used to
-        # updated rather than create objects)
-        if fields:
-            for field in list(self.fields.keys()):
-                if field not in fields:
-                    del self.fields[field]
+    def clean(self):
+        # Flag any invalid CSV headers
+        for header in self.headers:
+            if header not in self.fields:
+                raise forms.ValidationError(
+                    _("Unrecognized header: {name}").format(name=header)
+                )
+
+        return super().clean()
 
 
 class FilterForm(BootstrapMixin, forms.Form):


Test patch:
diff --git a/netbox/netbox/tests/test_import.py b/netbox/netbox/tests/test_import.py
--- a/netbox/netbox/tests/test_import.py
+++ b/netbox/netbox/tests/test_import.py
@@ -17,6 +17,36 @@ def setUpTestData(cls):
     def _get_csv_data(self, csv_data):
         return '\n'.join(csv_data)
 
+    def test_invalid_headers(self):
+        """
+        Test that import form validation fails when an unknown CSV header is present.
+        """
+        self.add_permissions('dcim.add_region')
+
+        csv_data = [
+            'name,slug,INVALIDHEADER',
+            'Region 1,region-1,abc',
+            'Region 2,region-2,def',
+            'Region 3,region-3,ghi',
+        ]
+        data = {
+            'format': ImportFormatChoices.CSV,
+            'data': self._get_csv_data(csv_data),
+            'csv_delimiter': CSVDelimiterChoices.AUTO,
+        }
+
+        # Form validation should fail with invalid header present
+        self.assertHttpStatus(self.client.post(self._get_url('import'), data), 200)
+        self.assertEqual(Region.objects.count(), 0)
+
+        # Correct the CSV header name
+        csv_data[0] = 'name,slug,description'
+        data['data'] = self._get_csv_data(csv_data)
+
+        # Validation should succeed
+        self.assertHttpStatus(self.client.post(self._get_url('import'), data), 302)
+        self.assertEqual(Region.objects.count(), 3)
+
     @override_settings(EXEMPT_VIEW_PERMISSIONS=['*'])
     def test_valid_tags(self):
         csv_data = (