Repository: cisagov/manage.get.gov

Issue:
Add a way in django admin to export/import data when not in production
### Issue description

With our analytics page and reports we often have issues with the large amount of data on stable and the limited amount of data on sandboxes. This means bugs get past testing environments and cause major errors on stable. If we could export table data from stable and import on developer sandboxes (as well as staging and development), then we could find errors faster and test additions that may not function well on large data sets.

We should allow exports from django admin in production, but do not allow import as an option for our production system

 **goal**
to easily have a non production system that matches production for ease of testing and for better early bug detection. Hence exporting data from stable and then importing it into other sandboxes will be the solution. 

**other options considered**
We do have the option of making a migration script that can load more fake data and go that route, but it will be fake data and isn't guaranteed to match stable closely enough to find the same level of errors. 

If the developer deems, after the  **time box of 5 days**, that this is not worth further dev time they can let the engineering team know and alternatives can be reconsidered

### Acceptance criteria. 

- [ ] tables that need this feature: user domain role, user, domain, domain info, domain request
- [ ] all tables (referred to above) in django admin have import and export buttons if not in production. In production all these tables only have export buttons
- [ ] importing should accept a file as an input and then make records for that table based on the values in the provided file
    - [ ] ideally it should not make duplicate entries nor should it throw an error if there is a value in the file that already exists in the table (for instance, if we want to make sure staging is somewhat up to date with stable, we would want to just import stables current file and have it add just the new values. Updating old values that have changed is a nice to have, but not needed as some discrepancy is expected between sandboxes)
- [ ] exporting- should just download the tables full contents with the sole purpose of being used for importing (don't worry about sorting values)

- [ ] when exporting data, emails should be replaced with fake data to avoid risking emails ever being sent to real people.
- [ ] Time box this at no more than 5 working days of time. If it's deemed to be overly complex call it out in dev channel. We can then evaluate if this idea should be scrapped and tickets made for alternative solutions (such as migration scripts)


### Additional context
**Suggestion** 
you may want to consider using a library such as [django-import-export ](https://django-import-export.readthedocs.io/en/latest/getting_started.html) to save time.  

**Further info, on the time box**
This time box was added per eng huddle meeting, when we discussed we could use scripts and given there are a lot of unknowns here. If you determine it's doable but better to start with a smaller number than 5 of the tables, that would be reasonable as a first step and we can make follow on tickets.

**Further info on overall end purpose:**
I would imagine once this is done, devs could download all the tables files from stable. Then add each file one by one to their sandbox. this would just be a one off task that they would then only repeat if they really needed to. If it's easier to just have one all encompassing import button and one all incompassing export button, that should work as well, it doesn't _have_ to be a button on each page. The benefit of it being on each page is if someone just wants to update a couple tables then they don't have to wait for all database data to download

**other**
we already have an environment variable for is_production, be sure to use this for disabling the import function

### Links to other issues

_No response_


Discussion:

Resolving patch:
diff --git a/src/registrar/admin.py b/src/registrar/admin.py
--- a/src/registrar/admin.py
+++ b/src/registrar/admin.py
@@ -7,7 +7,7 @@
 from django.db.models.functions import Concat, Coalesce
 from django.http import HttpResponseRedirect
 from django.shortcuts import redirect
-from django_fsm import get_available_FIELD_transitions
+from django_fsm import get_available_FIELD_transitions, FSMField
 from django.contrib import admin, messages
 from django.contrib.auth.admin import UserAdmin as BaseUserAdmin
 from django.contrib.auth.models import Group
@@ -30,12 +30,63 @@
 from django.utils.html import escape
 from django.contrib.auth.forms import UserChangeForm, UsernameField
 from django_admin_multiple_choice_list_filter.list_filters import MultipleChoiceListFilter
+from import_export import resources
+from import_export.admin import ImportExportModelAdmin
 
 from django.utils.translation import gettext_lazy as _
 
 logger = logging.getLogger(__name__)
 
 
+class FsmModelResource(resources.ModelResource):
+    """ModelResource is extended to support importing of tables which
+    have FSMFields.  ModelResource is extended with the following changes
+    to existing behavior:
+    When new objects are to be imported, FSMFields are initialized before
+    the object is initialized.  This is because FSMFields do not allow
+    direct modification.
+    When objects, which are to be imported, are updated, the FSMFields
+    are skipped."""
+
+    def init_instance(self, row=None):
+        """Overrides the init_instance method of ModelResource.  Returns
+        an instance of the model, with the FSMFields already initialized
+        from data in the row."""
+
+        # Get fields which are fsm fields
+        fsm_fields = {}
+
+        for f in self._meta.model._meta.fields:
+            if isinstance(f, FSMField):
+                if row and f.name in row:
+                    fsm_fields[f.name] = row[f.name]
+
+        # Initialize model instance with fsm_fields
+        return self._meta.model(**fsm_fields)
+
+    def import_field(self, field, obj, data, is_m2m=False, **kwargs):
+        """Overrides the import_field method of ModelResource.  If the
+        field being imported is an FSMField, it is not imported."""
+
+        is_fsm = False
+
+        # check each field in the object
+        for f in obj._meta.fields:
+            # if the field is an instance of FSMField
+            if field.attribute == f.name and isinstance(f, FSMField):
+                is_fsm = True
+        if not is_fsm:
+            super().import_field(field, obj, data, is_m2m, **kwargs)
+
+
+class UserResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.User
+
+
 class MyUserAdminForm(UserChangeForm):
     """This form utilizes the custom widget for its class's ManyToMany UIs.
 
@@ -498,9 +549,11 @@ class UserContactInline(admin.StackedInline):
     model = models.Contact
 
 
-class MyUserAdmin(BaseUserAdmin):
+class MyUserAdmin(BaseUserAdmin, ImportExportModelAdmin):
     """Custom user admin class to use our inlines."""
 
+    resource_classes = [UserResource]
+
     form = MyUserAdminForm
 
     class Meta:
@@ -706,17 +759,52 @@ class HostIPInline(admin.StackedInline):
     model = models.HostIP
 
 
-class MyHostAdmin(AuditedAdmin):
+class HostResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.Host
+
+
+class MyHostAdmin(AuditedAdmin, ImportExportModelAdmin):
     """Custom host admin class to use our inlines."""
 
+    resource_classes = [HostResource]
+
     search_fields = ["name", "domain__name"]
     search_help_text = "Search by domain or host name."
     inlines = [HostIPInline]
 
 
-class ContactAdmin(ListHeaderAdmin):
+class HostIpResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.HostIP
+
+
+class HostIpAdmin(AuditedAdmin, ImportExportModelAdmin):
+    """Custom host ip admin class"""
+
+    resource_classes = [HostIpResource]
+    model = models.HostIP
+
+
+class ContactResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.Contact
+
+
+class ContactAdmin(ListHeaderAdmin, ImportExportModelAdmin):
     """Custom contact admin class to add search."""
 
+    resource_classes = [ContactResource]
+
     search_fields = ["email", "first_name", "last_name"]
     search_help_text = "Search by first name, last name or email."
     list_display = [
@@ -837,9 +925,19 @@ def change_view(self, request, object_id, form_url="", extra_context=None):
         return super().change_view(request, object_id, form_url, extra_context=extra_context)
 
 
-class WebsiteAdmin(ListHeaderAdmin):
+class WebsiteResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.Website
+
+
+class WebsiteAdmin(ListHeaderAdmin, ImportExportModelAdmin):
     """Custom website admin class."""
 
+    resource_classes = [WebsiteResource]
+
     # Search
     search_fields = [
         "website",
@@ -887,9 +985,19 @@ def response_change(self, request, obj):
         return response
 
 
-class UserDomainRoleAdmin(ListHeaderAdmin):
+class UserDomainRoleResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.UserDomainRole
+
+
+class UserDomainRoleAdmin(ListHeaderAdmin, ImportExportModelAdmin):
     """Custom user domain role admin class."""
 
+    resource_classes = [UserDomainRoleResource]
+
     class Meta:
         """Contains meta information about this class"""
 
@@ -970,9 +1078,19 @@ class Meta:
     change_form_template = "django/admin/email_clipboard_change_form.html"
 
 
-class DomainInformationAdmin(ListHeaderAdmin):
+class DomainInformationResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.DomainInformation
+
+
+class DomainInformationAdmin(ListHeaderAdmin, ImportExportModelAdmin):
     """Customize domain information admin class."""
 
+    resource_classes = [DomainInformationResource]
+
     form = DomainInformationAdminForm
 
     # Columns
@@ -1103,9 +1221,19 @@ def get_readonly_fields(self, request, obj=None):
         return readonly_fields  # Read-only fields for analysts
 
 
-class DomainRequestAdmin(ListHeaderAdmin):
+class DomainRequestResource(FsmModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.DomainRequest
+
+
+class DomainRequestAdmin(ListHeaderAdmin, ImportExportModelAdmin):
     """Custom domain requests admin class."""
 
+    resource_classes = [DomainRequestResource]
+
     form = DomainRequestAdminForm
     change_form_template = "django/admin/domain_request_change_form.html"
 
@@ -1648,9 +1776,19 @@ def get_readonly_fields(self, request, obj=None):
         return DomainInformationAdmin.get_readonly_fields(self, request, obj=None)
 
 
-class DomainAdmin(ListHeaderAdmin):
+class DomainResource(FsmModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.Domain
+
+
+class DomainAdmin(ListHeaderAdmin, ImportExportModelAdmin):
     """Custom domain admin class to add extra buttons."""
 
+    resource_classes = [DomainResource]
+
     class ElectionOfficeFilter(admin.SimpleListFilter):
         """Define a custom filter for is_election_board"""
 
@@ -2045,9 +2183,19 @@ def has_change_permission(self, request, obj=None):
         return super().has_change_permission(request, obj)
 
 
-class DraftDomainAdmin(ListHeaderAdmin):
+class DraftDomainResource(resources.ModelResource):
+    """defines how each field in the referenced model should be mapped to the corresponding fields in the
+    import/export file"""
+
+    class Meta:
+        model = models.DraftDomain
+
+
+class DraftDomainAdmin(ListHeaderAdmin, ImportExportModelAdmin):
     """Custom draft domain admin class."""
 
+    resource_classes = [DraftDomainResource]
+
     search_fields = ["name"]
     search_help_text = "Search by draft domain name."
 
@@ -2183,9 +2331,8 @@ class Meta:
 admin.site.register(models.Domain, DomainAdmin)
 admin.site.register(models.DraftDomain, DraftDomainAdmin)
 admin.site.register(models.FederalAgency, FederalAgencyAdmin)
-# Host and HostIP removed from django admin because changes in admin
-# do not propagate to registry and logic not applied
 admin.site.register(models.Host, MyHostAdmin)
+admin.site.register(models.HostIP, HostIpAdmin)
 admin.site.register(models.Website, WebsiteAdmin)
 admin.site.register(models.PublicContact, PublicContactAdmin)
 admin.site.register(models.DomainRequest, DomainRequestAdmin)
diff --git a/src/registrar/config/settings.py b/src/registrar/config/settings.py
--- a/src/registrar/config/settings.py
+++ b/src/registrar/config/settings.py
@@ -147,6 +147,8 @@
     "corsheaders",
     # library for multiple choice filters in django admin
     "django_admin_multiple_choice_list_filter",
+    # library for export and import of data
+    "import_export",
     # Waffle feature flags
     "waffle",
 ]
diff --git a/src/registrar/models/utility/generic_helper.py b/src/registrar/models/utility/generic_helper.py
--- a/src/registrar/models/utility/generic_helper.py
+++ b/src/registrar/models/utility/generic_helper.py
@@ -100,36 +100,41 @@ def _handle_new_instance(self):
         # Update the field
         self._update_fields(organization_type_needs_update, generic_org_type_needs_update)
 
-    def _handle_existing_instance(self, force_update_when_no_are_changes_found=False):
+    def _handle_existing_instance(self, force_update_when_no_changes_are_found=False):
         # == Init variables == #
-        # Instance is already in the database, fetch its current state
-        current_instance = self.sender.objects.get(id=self.instance.id)
-
-        # Check the new and old values
-        generic_org_type_changed = self.instance.generic_org_type != current_instance.generic_org_type
-        is_election_board_changed = self.instance.is_election_board != current_instance.is_election_board
-        organization_type_changed = self.instance.organization_type != current_instance.organization_type
-
-        # == Check for invalid conditions before proceeding == #
-        if organization_type_changed and (generic_org_type_changed or is_election_board_changed):
-            # Since organization type is linked with generic_org_type and election board,
-            # we have to update one or the other, not both.
-            # This will not happen in normal flow as it is not possible otherwise.
-            raise ValueError("Cannot update organization_type and generic_org_type simultaneously.")
-        elif not organization_type_changed and (not generic_org_type_changed and not is_election_board_changed):
-            # No changes found
-            if force_update_when_no_are_changes_found:
-                # If we want to force an update anyway, we can treat this record like
-                # its a new one in that we check for "None" values rather than changes.
-                self._handle_new_instance()
-        else:
-            # == Update the linked values == #
-            # Find out which field needs updating
-            organization_type_needs_update = generic_org_type_changed or is_election_board_changed
-            generic_org_type_needs_update = organization_type_changed
-
-            # Update the field
-            self._update_fields(organization_type_needs_update, generic_org_type_needs_update)
+        try:
+            # Instance is already in the database, fetch its current state
+            current_instance = self.sender.objects.get(id=self.instance.id)
+
+            # Check the new and old values
+            generic_org_type_changed = self.instance.generic_org_type != current_instance.generic_org_type
+            is_election_board_changed = self.instance.is_election_board != current_instance.is_election_board
+            organization_type_changed = self.instance.organization_type != current_instance.organization_type
+
+            # == Check for invalid conditions before proceeding == #
+            if organization_type_changed and (generic_org_type_changed or is_election_board_changed):
+                # Since organization type is linked with generic_org_type and election board,
+                # we have to update one or the other, not both.
+                # This will not happen in normal flow as it is not possible otherwise.
+                raise ValueError("Cannot update organization_type and generic_org_type simultaneously.")
+            elif not organization_type_changed and (not generic_org_type_changed and not is_election_board_changed):
+                # No changes found
+                if force_update_when_no_changes_are_found:
+                    # If we want to force an update anyway, we can treat this record like
+                    # its a new one in that we check for "None" values rather than changes.
+                    self._handle_new_instance()
+            else:
+                # == Update the linked values == #
+                # Find out which field needs updating
+                organization_type_needs_update = generic_org_type_changed or is_election_board_changed
+                generic_org_type_needs_update = organization_type_changed
+
+                # Update the field
+                self._update_fields(organization_type_needs_update, generic_org_type_needs_update)
+        except self.sender.DoesNotExist:
+            # this exception should only be raised when import_export utility attempts to import
+            # a new row and already has an id
+            pass
 
     def _update_fields(self, organization_type_needs_update, generic_org_type_needs_update):
         """


Test patch:
diff --git a/src/registrar/tests/test_admin.py b/src/registrar/tests/test_admin.py
--- a/src/registrar/tests/test_admin.py
+++ b/src/registrar/tests/test_admin.py
@@ -21,6 +21,7 @@
     MyHostAdmin,
     UserDomainRoleAdmin,
     VerifiedByStaffAdmin,
+    FsmModelResource,
     WebsiteAdmin,
     DraftDomainAdmin,
     FederalAgencyAdmin,
@@ -61,7 +62,7 @@
 )
 from django.contrib.sessions.backends.db import SessionStore
 from django.contrib.auth import get_user_model
-from unittest.mock import ANY, call, patch
+from unittest.mock import ANY, call, patch, Mock
 from unittest import skip
 
 from django.conf import settings
@@ -71,6 +72,49 @@
 logger = logging.getLogger(__name__)
 
 
+class TestFsmModelResource(TestCase):
+    def setUp(self):
+        self.resource = FsmModelResource()
+
+    def test_init_instance(self):
+        """Test initializing an instance of a class with a FSM field"""
+
+        # Mock a row with FSMField data
+        row_data = {"state": "ready"}
+
+        self.resource._meta.model = Domain
+
+        instance = self.resource.init_instance(row=row_data)
+
+        # Assert that the instance is initialized correctly
+        self.assertIsInstance(instance, Domain)
+        self.assertEqual(instance.state, "ready")
+
+    def test_import_field(self):
+        """Test that importing a field does not import FSM field"""
+
+        # Mock a FSMField and a non-FSM-field
+        fsm_field_mock = Mock(attribute="state", column_name="state")
+        field_mock = Mock(attribute="name", column_name="name")
+        # Mock the data
+        data_mock = {"state": "unknown", "name": "test"}
+        # Define a mock Domain
+        obj = Domain(state=Domain.State.UNKNOWN, name="test")
+
+        # Mock the save() method of fields so that we can test if save is called
+        # save() is only supposed to be called for non FSM fields
+        field_mock.save = Mock()
+        fsm_field_mock.save = Mock()
+
+        # Call the method with FSMField and non-FSMField
+        self.resource.import_field(fsm_field_mock, obj, data=data_mock, is_m2m=False)
+        self.resource.import_field(field_mock, obj, data=data_mock, is_m2m=False)
+
+        # Assert that field.save() in super().import_field() is called only for non-FSMField
+        field_mock.save.assert_called_once()
+        fsm_field_mock.save.assert_not_called()
+
+
 class TestDomainAdmin(MockEppLib, WebTest):
     # csrf checks do not work with WebTest.
     # We disable them here. TODO for another ticket.
@@ -600,10 +644,9 @@ def test_short_org_name_in_domains_list(self):
                 domain_request.approve()
 
             response = self.client.get("/admin/registrar/domain/")
-
             # There are 4 template references to Federal (4) plus four references in the table
             # for our actual domain_request
-            self.assertContains(response, "Federal", count=42)
+            self.assertContains(response, "Federal", count=54)
             # This may be a bit more robust
             self.assertContains(response, '<td class="field-generic_org_type">Federal</td>', count=1)
             # Now let's make sure the long description does not exist
@@ -852,6 +895,14 @@ def test_analyst_deletes_domain_idempotent(self):
     def test_place_and_remove_hold_epp(self):
         raise
 
+    @override_settings(IS_PRODUCTION=True)
+    def test_prod_only_shows_export(self):
+        """Test that production environment only displays export"""
+        with less_console_noise():
+            response = self.client.get("/admin/registrar/domain/")
+            self.assertContains(response, ">Export<")
+            self.assertNotContains(response, ">Import<")
+
     def tearDown(self):
         super().tearDown()
         PublicContact.objects.all().delete()
@@ -1376,7 +1427,7 @@ def test_short_org_name_in_domain_requests_list(self):
             response = self.client.get("/admin/registrar/domainrequest/?generic_org_type__exact=federal")
             # There are 2 template references to Federal (4) and two in the results data
             # of the request
-            self.assertContains(response, "Federal", count=40)
+            self.assertContains(response, "Federal", count=52)
             # This may be a bit more robust
             self.assertContains(response, '<td class="field-generic_org_type">Federal</td>', count=1)
             # Now let's make sure the long description does not exist