Repository: netbox-community/netbox
Pull request: CSV file import
### Fixes: #6560

An example / first draft of a solution for #6560 .

In the current implementation, if users try to use both the `CSVFileField` and the `CSVDataField`, the `CSVDataField` will be ignored and only the objects in the file will be imported. It should be unusual that both fields are populated, but let me know if you prefer to handle that case differently, such as by raising a validation error. The csv file is checked for errors (incorrect formatting, objects already exist, invalid columns) before importing.

This solution involves some duplication between `CSVFileField` and `CSVDataField`. If you prefer, I could remove most of the duplication by refactoring them to both inherit from a single class - say, `CSVParsingField`.


Problem statement:
CSV file import
### NetBox version

v2.11.6

### Feature type

New functionality

### Proposed functionality

Allows users to upload a csv file of new objects into netbox in addition to the current textbox for bulk import. 

### Use case

Provides an easier way for users to upload large amounts of data into netbox at one time. 

### Database changes

_No response_

### External dependencies

_No response_

Hints from issue discussion:

Commit message: removed unnecessary use of seek()

Patch to review:
diff --git a/netbox/netbox/views/generic.py b/netbox/netbox/views/generic.py
index 8a2bae4fa3b..24d4592199e 100644
--- a/netbox/netbox/views/generic.py
+++ b/netbox/netbox/views/generic.py
@@ -20,7 +20,7 @@
 from utilities.error_handlers import handle_protectederror
 from utilities.exceptions import AbortTransaction
 from utilities.forms import (
-    BootstrapMixin, BulkRenameForm, ConfirmationForm, CSVDataField, ImportForm, TableConfigForm, restrict_form_fields,
+    BootstrapMixin, BulkRenameForm, ConfirmationForm, CSVDataField, ImportForm, TableConfigForm, restrict_form_fields, CSVFileField
 )
 from utilities.permissions import get_permission_for_model
 from utilities.tables import paginate_table
@@ -665,7 +665,10 @@ class ImportForm(BootstrapMixin, Form):
                 from_form=self.model_form,
                 widget=Textarea(attrs=self.widget_attrs)
             )
-
+            upload_csv = CSVFileField(
+                from_form=self.model_form,
+                required=False
+            )
         return ImportForm(*args, **kwargs)
 
     def _save_obj(self, obj_form, request):
@@ -689,7 +692,7 @@ def get(self, request):
     def post(self, request):
         logger = logging.getLogger('netbox.views.BulkImportView')
         new_objs = []
-        form = self._import_form(request.POST)
+        form = self._import_form(request.POST, request.FILES)
 
         if form.is_valid():
             logger.debug("Form validation was successful")
@@ -697,7 +700,10 @@ def post(self, request):
             try:
                 # Iterate through CSV data and bind each row to a new model form instance.
                 with transaction.atomic():
-                    headers, records = form.cleaned_data['csv']
+                    if request.FILES:
+                        headers, records = form.cleaned_data['upload_csv']
+                    else:
+                        headers, records = form.cleaned_data['csv']
                     for row, data in enumerate(records, start=1):
                         obj_form = self.model_form(data, headers=headers)
                         restrict_form_fields(obj_form, request.user)
diff --git a/netbox/templates/generic/object_bulk_import.html b/netbox/templates/generic/object_bulk_import.html
index 170cf366554..e9972e9196c 100644
--- a/netbox/templates/generic/object_bulk_import.html
+++ b/netbox/templates/generic/object_bulk_import.html
@@ -20,7 +20,7 @@ <h1>{% block title %}{{ obj_type|bettertitle }} Bulk Import{% endblock %}</h1>
             </ul>
             <div class="tab-content">
                 <div role="tabpanel" class="tab-pane active" id="csv">
-                    <form action="" method="post" class="form">
+                    <form action="" method="post" class="form" enctype="multipart/form-data">
                         {% csrf_token %}
                         {% render_form form %}
                         <div class="form-group">
diff --git a/netbox/utilities/forms/fields.py b/netbox/utilities/forms/fields.py
index 93b9e6c44d0..0f5b0b6dd45 100644
--- a/netbox/utilities/forms/fields.py
+++ b/netbox/utilities/forms/fields.py
@@ -26,6 +26,7 @@
     'CSVChoiceField',
     'CSVContentTypeField',
     'CSVDataField',
+    'CSVFileField',
     'CSVModelChoiceField',
     'CSVTypedChoiceField',
     'DynamicModelChoiceField',
@@ -221,6 +222,77 @@ def validate(self, value):
         return value
 
 
+class CSVFileField(forms.FileField):
+    """
+    A FileField (rendered as a file input button) which accepts a file containing CSV-formatted data. It returns
+    data as a two-tuple: The first item is a dictionary of column headers, mapping field names to the attribute
+    by which they match a related object (where applicable). The second item is a list of dictionaries, each
+    representing a discrete row of CSV data.
+
+    :param from_form: The form from which the field derives its validation rules.
+    """
+
+    def __init__(self, from_form, *args, **kwargs):
+
+        form = from_form()
+        self.model = form.Meta.model
+        self.fields = form.fields
+        self.required_fields = [
+            name for name, field in form.fields.items() if field.required
+        ]
+
+        super().__init__(*args, **kwargs)
+
+    def to_python(self, file):
+
+        records = []
+        csv_str = file.read().decode('utf-8')
+        reader = csv.reader(csv_str.splitlines())
+
+        # Consume the first line of CSV data as column headers. Create a dictionary mapping each header to an optional
+        # "to" field specifying how the related object is being referenced. For example, importing a Device might use a
+        # `site.slug` header, to indicate the related site is being referenced by its slug.
+
+        headers = {}
+        for header in next(reader):
+            if '.' in header:
+                field, to_field = header.split('.', 1)
+                headers[field] = to_field
+            else:
+                headers[header] = None
+
+        # Parse CSV rows into a list of dictionaries mapped from the column headers.
+        for i, row in enumerate(reader, start=1):
+            if len(row) != len(headers):
+                raise forms.ValidationError(
+                    f"Row {i}: Expected {len(headers)} columns but found {len(row)}"
+                )
+            row = [col.strip() for col in row]
+            record = dict(zip(headers.keys(), row))
+            records.append(record)
+
+        return headers, records
+
+    def validate(self, value):
+        headers, records = value
+
+        # Validate provided column headers
+        for field, to_field in headers.items():
+            if field not in self.fields:
+                raise forms.ValidationError(f'Unexpected column header "{field}" found.')
+            if to_field and not hasattr(self.fields[field], 'to_field_name'):
+                raise forms.ValidationError(f'Column "{field}" is not a related object; cannot use dots')
+            if to_field and not hasattr(self.fields[field].queryset.model, to_field):
+                raise forms.ValidationError(f'Invalid related object attribute for column "{field}": {to_field}')
+
+        # Validate required fields
+        for f in self.required_fields:
+            if f not in headers:
+                raise forms.ValidationError(f'Required column header "{f}" not found.')
+
+        return value
+
+
 class CSVChoiceField(forms.ChoiceField):
     """
     Invert the provided set of choices to take the human-friendly label as input, and return the database value.


Code review comments:
diff_hunk: @@ -221,6 +222,77 @@ def validate(self, value):
         return value
 
 
+class CSVFileField(forms.FileField):
path: netbox/utilities/forms/fields.py
text: @user1:
Most of the content of this class seems to have been duplicated from CSVDataField: We always want to minimize redundant code. Instead of duplicating the logic, we should move it out of CSVDataField into a separate class or functions that can be utilized by both CSVDataField and CSVFileField.

@author:
I made two functions, parse_csv and validate_csv, within forms/utils.py. Both CSV fields use these functions to reduce redundancy.

diff_hunk: @@ -665,7 +665,10 @@ class ImportForm(BootstrapMixin, Form):
                 from_form=self.model_form,
                 widget=Textarea(attrs=self.widget_attrs)
             )
-
+            upload_csv = CSVFileField(
path: netbox/netbox/views/generic.py
text: @user1:
This field needs a human-friendly label (e.g. "CSV file")

@user1:
Also, `csv_file` might be a better name than `upload_csv`.

@author:
Renamed.