Repository: pulp/pulpcore

Issue:
An import of resources fails because of an unsaved casted object
Reproducible in pulp_container:
> src/pulp_container/pulp_container/tests/functional/api/test_pulpimportexport.py::test_import_export_create_repositories 


After merging the commit https://github.com/pulp/pulpcore/commit/977a04dbe32d8004629713ffe3121702662bc640, we started to experience runtime errors when importing repositories which were not created in advance.

```
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.app.tasks.importer:INFO: ...Importing resource BlobResource.
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.app.tasks.importer:INFO: Importing file ./tmpkyf64nd0/repository-18fa8187-5f8f-4731-a486-b997a495ddcc_1/pulp_container.app.modelresource.ManifestResource.json.
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.app.tasks.importer:INFO: ...Importing resource ManifestResource.
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.app.tasks.importer:INFO: ...1 import-errors encountered importing ./tmpkyf64nd0/repository-18fa8187-5f8f-4731-a486-b997a495ddcc_1/pulp_container.app.modelresource.ManifestResource.json, attempt 2, retrying
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.app.tasks.importer:INFO: ...1 import-errors encountered importing ./tmpkyf64nd0/repository-18fa8187-5f8f-4731-a486-b997a495ddcc_1/pulp_container.app.modelresource.ManifestResource.json, attempt 3, retrying
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.app.tasks.importer:ERROR: FATAL import-failure importing ./tmpkyf64nd0/repository-18fa8187-5f8f-4731-a486-b997a495ddcc_1/pulp_container.app.modelresource.ManifestResource.json
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.tasking.tasks:INFO: Task 01887a16-9b4b-7745-a946-19e15dcd84f6 failed (None)
pulp [0c7ff191-3103-4829-b68b-e88c40ed1d7e]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/tasks.py", line 66, in _execute_task
    result = func(*args, **kwargs)

  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/tasks/importer.py", line 236, in import_repository_version
    for a_result in _import_file(os.path.join(rv_path, filename), res_class, retry=True):

  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/tasks/importer.py", line 138, in _import_file
    a_result = resource.import_data(data, raise_errors=True)

  File "/usr/local/lib/python3.8/site-packages/import_export/resources.py", line 813, in import_data
    result = self.import_data_inner(

  File "/usr/local/lib/python3.8/site-packages/import_export/resources.py", line 882, in import_data_inner
    raise row_result.errors[-1].error

  File "/usr/local/lib/python3.8/site-packages/import_export/resources.py", line 722, in import_row
    diff = self.get_diff_class()(self, original, new)

  File "/usr/local/lib/python3.8/site-packages/import_export/resources.py", line 237, in __init__
    self.left = self._export_resource_fields(resource, instance)

  File "/usr/local/lib/python3.8/site-packages/import_export/resources.py", line 258, in _export_resource_fields
    return [resource.export_field(f, instance) if instance else "" for f in resource.get_user_visible_fields()]

  File "/usr/local/lib/python3.8/site-packages/import_export/resources.py", line 258, in <listcomp>
    return [resource.export_field(f, instance) if instance else "" for f in resource.get_user_visible_fields()]

  File "/usr/local/lib/python3.8/site-packages/import_export/resources.py", line 931, in export_field
    return field.export(obj)

  File "/usr/local/lib/python3.8/site-packages/import_export/fields.py", line 134, in export
    value = self.get_value(obj)

  File "/usr/local/lib/python3.8/site-packages/import_export/fields.py", line 96, in get_value
    value = getattr(value, attr, None)

  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 617, in __get__
    return self.related_manager_cls(instance)

  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line [1023](https://github.com/pulp/pulp_container/actions/runs/5151407496/jobs/9276547218#step:18:1024), in __init__
    '"%r" needs to have a value for field "%s" before '

  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/base.py", line 65, in __repr__
    return str(self)

  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/base.py", line 200, in __str__
    cast = self.cast()

  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/base.py", line 169, in cast
    result = self._pulp_model_map[self.pulp_type].objects.get(pk=self.pk)
```

Trapped logging:
```
 def cast(self):
        """Return the "Detail" model instance of this master-detail object.

        If this is already an instance of its detail type, it will return itself.
        """
        if self.pulp_type == self.get_pulp_type():
            return self
        try:
            result = self._pulp_model_map[self.pulp_type].objects.get(pk=self.pk)
        except Exception as e:
            raise RuntimeError("eeeeeeeeeeeeeeee: " + str(e) + f": {self.pulp_type}, {self.pk}, {self._pulp_model_map} \n {self.get_pulp_type()}")


pulp [c5a37660-a5f1-453e-ba96-a4ce6d01b4d9]: pulpcore.tasking.tasks:INFO: Task 01887c8b-78ce-77b4-921f-4d293330124c failed (eeeeeeeeeeeeeeee: None: None, None, {'core.content': <class 'pulpcore.app.models.content.Content'>, 'core.publishedmetadata': <class 'pulpcore.app.models.publication.PublishedMetadata'>, 'container.blob': <class 'pulp_container.app.models.Blob'>, 'container.manifest': <class 'pulp_container.app.models.Manifest'>, 'container.tag': <class 'pulp_container.app.models.Tag'>, 'container.signature': <class 'pulp_container.app.models.ManifestSignature'>} 
 container.manifest)


```


Discussion:
The problem is that the content is not yet present in the database and it fails on this `ValueError` exception because of the `%r` variable that invokes the `__str__` method that calls `cast` on `self.pulp_type=None`:
```
            # file: /usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py

            self.related_val = self.source_field.get_foreign_related_value(instance)
            if None in self.related_val:
                #++ raise IOError(f"{self.model} ++: {self.query_field_name} :++ {self.pk_field_names}")  -----------> (<class 'pulp_container.app.models.Blob'> ++: container_manifest :++ {'manifest': 'content_ptr'})
                raise ValueError(
                    '"%r" needs to have a value for field "%s" before '
                    "this many-to-many relationship can be used."
                    % (instance, self.pk_field_names[self.source_field_name])
                )
```

EDIT: we can try to create manifest-blob pairs without direct object references but using "PK placeholders". Not sure if it is worth the effort if everything fails just on the `__str__` method which invokes `cast()` that does not handle a situation where model instances are not saved. Moving to pulpcore.
We should not expect that the instance exists (when importing related content) and act based on that accordingly.

https://github.com/pulp/pulpcore/blob/456ebe447d686b77fadcfb9ee57462f692df8226/pulpcore/app/models/base.py#L198

#### self.pulp_type = None; self.get_pulp_type == container.manifest
https://github.com/pulp/pulpcore/blob/456ebe447d686b77fadcfb9ee57462f692df8226/pulpcore/app/models/base.py#L167
I suggest modifying `MasterModel.__str__`, e.g., like this:
```diff
    def __str__(self):
        # similar to Model's __str__, but type-aware
+        if self.pulp_type is None:
+            return super().__str__()
+
        cast = self.cast()
        if cast is self:
            return super().__str__()

        try:
            return "<{} (pulp_type={}): {}>".format(
                self._meta.object_name, cast.pulp_type, cast.name
            )
        except AttributeError:
            return "<{} (pulp_type={}): pk={}>".format(
                self._meta.object_name, cast.pulp_type, cast.pk
            )
```
I'd say it is an error to call cast in `__str__` at all. I would want to reduce it to:

```python
    def __str__(self):
        # similar to Model's __str__, but type-aware
        return "<{} (pulp_type={}): pk={}>".format(
            self._meta.object_name, self.pulp_type, self.pk
        )
```
And subclasses are free to overwrite it, so it looks pretty after casting. (Assuming that the name of a subclass makes for a good key was also a big mistake.)

Why does export/import call str on models in the first place?
Well, the Django internals for the related objects lookup contains the code which may return an exception with the instance's `__repr__`. The exception can be raised when importing content that does not exist yet in the database: https://github.com/pulp/pulpcore/issues/3898#issuecomment-1575611847

Resolving patch:
diff --git a/pulpcore/app/models/base.py b/pulpcore/app/models/base.py
--- a/pulpcore/app/models/base.py
+++ b/pulpcore/app/models/base.py
@@ -197,18 +197,7 @@ def master(self):
 
     def __str__(self):
         # similar to Model's __str__, but type-aware
-        cast = self.cast()
-        if cast is self:
-            return super().__str__()
-
-        try:
-            return "<{} (pulp_type={}): {}>".format(
-                self._meta.object_name, cast.pulp_type, cast.name
-            )
-        except AttributeError:
-            return "<{} (pulp_type={}): pk={}>".format(
-                self._meta.object_name, cast.pulp_type, cast.pk
-            )
+        return "<{} (pulp_type={}): pk={}>".format(self._meta.object_name, self.pulp_type, self.pk)
 
 
 # Add properties to model _meta info to support master/detail models


Test patch: