Repository: carpentries/amy

Issue:
Bulk import workflow encounters IntegrityError when saving an organization
Currently, we allow organizations with the domain that contains the `www` subdomain. For eg: Google can exist as `www.google.com` as well as `google.com`, leading to `IntegrityError` while saving the first while the second exists.

Shouldn't we enforce one URL pattern and trim/add `www` to the `domain` field when saving an organization?

Testcase:

``` py
In [5]: Organization.objects.create(fullname='Google', domain='google.com')
Out[5]: <Organization: google.com>

In [6]: Organization.objects.create(fullname='Google', domain='www.google.com')
---------------------------------------------------------------------------
IntegrityError                            Traceback (most recent call last)
```



Discussion:
Should we use the [tld](https://pypi.python.org/pypi/tld) package to fetch the top-level domain from a URL?

@pbanaszkiewicz ping

Does this integrity error happen when doing stuff from our UI? Because I do expect it to happen when you manually try to add a second organization with the same `fullname`.

The particular failing test case is:

`L122: pydata/api.py`

``` py
organization=Organization.objects.get_or_create(
                domain=urlparse(sponsor['external_url']).netloc,
                defaults={
                    'fullname': sponsor['name'],
                    'notes': sponsor['annotation'],
                },
            )[0]
```

The ambiguity around the `domain` field would then force me to search for organizations with `domain='google.com'` and `domain='www.google.com'`, since I am not sure what the admin used while manually saving it using the UI.
I can filter organizations by their `fullname` but that can be problematic.

You need to look up `Organization` by both `domain` and `fullname` since the uniqueness constraint is on both independently.

No, I'm wrong, you should look up by `fullname` and assume that:
1. if the org with this `fullname` exists, it has correct `domain` (otherwise you should update)
2. if the org doesn't exist, create with domain from external url

Aren't we putting too much faith in a `CharField` to unique identify an organization?

``` py
In [3]: Organization.objects.create(fullname='European Organization for Nuclear Research', domain='home.cern')
Out[3]: <Organization: home.cern>

In [4]: Organization.objects.get(fullname='CERN')
# Throws DoesNotExist
```

I'd say we're shifting the burden of identifying organization by it's name from AMY to human (because humans are intelligent and AMY is not).

That throws problems when we want AMY to do it; like during the bulk import. There is no human intervention here that can help AMY uniquely identify an organization that can have different names and different domains at different conference sites.

I propose we exclusively use the `domain` field to uniquely identify organizations after we enforce a strict domain convention (no `www` subdomain). `tld` package can help us here.

> I propose we exclusively use the domain field to uniquely identify organizations after we enforce a strict domain convention (no www subdomain). tld package can help us here.

-1 - I've seen a number of places that serve content only with `www.` subdomain… Essentially, `www.domain.tld` != `domain.tld`.

And I believe there exist organizations that serve different content on the root domain and the `www` subdomain.
Am I out of luck here?

I think I will have to resort to using `Q` objects to conditionally lookup organizations based on both fields: `domain` and `fullname`.

> And I believe there exist organizations that serve different content on the root domain and the www subdomain.

Like… organizations that we have in the DB?

> Like… organizations that we have in the DB?

I cannot say for sure. I just wanted to say that `www.domain.tld != domain.tld` is true both ways. We cannot strip `www` out of domains and neither can we prepend `www` to domains.

@narayanaditya95 should we close this issue or are you perhaps going to make some changes in the code that refer to our discussion?

I will rewrite the query to the code I referred.

To be true, I still do not have a fool-proof solution that allows me to uniquely identify organizations that can exist with slightly different names and domains.

When you say that organizations do not necessarily serve on the root domain (`www.domain.tld` != `domain.tld`), and would rather serve on the `www` subdomain, can we assume that there always exist a redirect from the root domain to the `www` subdomain?
For eg, if we fetch an organization with the domain `www.example.org`; can we be sure that `example.org` will always redirect to `www.example.org`. This does not necessarily imply that if we have fetched an organization with the domain `example.org`, the subdomain `www.example.org` will be serving content.

Is that a sane argument in favour of having to filter using the query `Q(domain=domain) | Q(domain=domain.lstrip('www.')`?

@narayanaditya95 to be honest, I never liked doing such pseudo-AI matching and deciding on that. If I were you, I'd add a tool for merging organizations and show user an info "There are some similar orgs in the database, make sure to run the merging tool if they're the same".

@chrismedrela Can you pitch in?

@narayanaditya95 Why is it an issue that we don't check for www subdomain in organization domain? Is it the risk of duplicating records for the same organization? If yes, is adding one single organization the only way we can introduce duplicates (or do we have any bulk import organization functionality)?

The problem is that due to two different unique constraints (each on the `fullname` and `domain` fields), I cannot check if a similar org exists, before saving a new one. If I filter based on the `domain` field, I can get into a uniqueness failure on the `fullname` field and vice versa.

Checking for www subdomain is one fix to the problem. The other fix is to convert the individual unique constraints into a unqiue_together constraint.

> I cannot check if a similar org exists, before saving a new one

``` python
Organization.objects.filter(Q(fullname=fullname) | Q(domain=domain)).first() # returns one record or None
```

Does it solve the issue? If no, why?

Thank you. :)

Can we close it or is there anything else that need to solved?

There has to be a fix for the bulk import workflow.

@narayanaditya95 , could the issue with bulk import workflow go to a new issue and we close this one, or could you update the title of this issue and provide short description of the issue with bulk import workflow? 

@gvwilson, do we want to fix the bulk import issue in v1.9 or can we postpone it to v1.10?

Too late for v1.9 - I was hoping that would be deployed this weekend.


Resolving patch:
diff --git a/pydata/api.py b/pydata/api.py
--- a/pydata/api.py
+++ b/pydata/api.py
@@ -4,6 +4,7 @@
 
 import requests
 from django.conf import settings
+from django.db.models import Q
 
 from workshops.models import (
     Person,
@@ -118,14 +119,18 @@ class SponsorshipAPIClient(BaseAPIClient):
     model = Sponsorship
 
     def parse(self, sponsor):
+        domain = urlparse(sponsor['external_url']).netloc
+        organization = Organization.objects.filter(
+            Q(fullname=sponsor['name']) | Q(domain=domain)
+        ).first()
+        if not organization:
+            organization = Organization.objects.create(
+                fullname=sponsor['name'],
+                domain=domain,
+                notes=sponsor['annotation'],
+            )
         return Sponsorship(
-            organization=Organization.objects.get_or_create(
-                domain=urlparse(sponsor['external_url']).netloc,
-                defaults={
-                    'fullname': sponsor['name'],
-                    'notes': sponsor['annotation'],
-                },
-            )[0],
+            organization=organization,
             event=self.event,
             amount=sponsor['level']['cost'],
             contact=Person.objects.get_or_create(


Test patch: