Repository: hydroshare/hydroshare

Issue:
add a select for "subject areas"
**Describe the feature you'd like and what it will do**
Currently, subject areas is free text.
Should be a typeahead -- noninvasive.
@martinseul suggested that we should populate a list similar to
https://has.arizona.edu/research-focus-areas


Related to https://github.com/hydroshare/hydroshare/issues/4733



Discussion:
@martinseul who might be best to review our list of [proposed subject areas](https://github.com/hydroshare/hydroshare/blob/5022-subject-areas/hs_dictionary/management/commands/subject_areas.txt)?
@dtarb here is our list of [proposed subject areas](https://github.com/hydroshare/hydroshare/blob/5022-subject-areas/hs_dictionary/management/commands/subject_areas.txt). Do you (or anyone else appropriate) have any comments?
It came from here: https://has.arizona.edu/research-focus-areas

This list populates a typahead on the "Subject Areas" of our Profile Page
I'm going to punt on this.  I can not do everything.  Sorry if this comes off snarky, but it is late at night.  We used to have the concept of product owners as people who owned different areas of the product.  The document may be dated but look at [this](https://docs.google.com/document/d/18T1Sq390jm7vbH3Xg6nAZuGpD0lV9KxVoGDcTJGfQCM/edit?usp=sharing).  Product owner for user profile was either @ClaraCogswell or @Castronova .  Part of the reason for punting on this is that I do not fully appreciate what we are trying to achieve by having a type ahead list for this.  It would not be high on my priorities, or even an issue in our list of bugs and backlog to address.  We have much bigger issues to address like the scalability problem of not being able to add to a collection.
I'll take it. @devincowan  this list looks good, as long as we also make it clear that they can enter something that is not in the type ahead. I've gotten questions about this before "do I have to use something from the list my thing is not on the list", so we should have a little note somewhere that says it is free text entry, the type ahead is just for convenience/search efficiency but there is not actually a controlled vocab being enforced (unless there is but I thought I remembered that not being hte case). 
My $0.02 - I'm with Clara. The list looks mostly fine as a type-ahead list of suggestions. There's some really long and specific ones in there, though, that don't seem that broadly useful. We don't want to force people to select specific terms from this list unless they want to.
Thanks @dtarb for the product owners list -- I hadn't seen this, sorry for the hassle. I agree we have bigger fish to fry, so I'll try not to linger here.
@ClaraCogswell noted -- adding a note to make it obvious that free text is permissible.
And @horsburgh thanks for the cents!
Improving the profile data and the ability to report on users is important to CUAHSI and the additional lift for adding the subject type ahead while fixing the regression we discovered for the institutions is a small lift so IMHO worth the effort. 
I agree on looking at the product owners to help distribute work. The list is a bit outdated as you mentioned and we should revisit, maybe a topic for the HS governance discussion.

Resolving patch:
diff --git a/theme/migrations/0023_alter_userprofile_subject_areas.py b/theme/migrations/0023_alter_userprofile_subject_areas.py
--- a/theme/migrations/0023_alter_userprofile_subject_areas.py
+++ b/theme/migrations/0023_alter_userprofile_subject_areas.py
@@ -2,12 +2,25 @@
 
 import django.contrib.postgres.fields
 from django.db import migrations, models
+from django.db.utils import DataError
+from django.core.management import call_command
+import re
 
 
 def migrate_csv_subject_areas(apps, schema_editor):
+    def strip_for_dict(string=""):
+        # Check if the string consists of only spaces and braces
+        res, _ = re.subn('{|}', '', string)
+        if res.strip() == "":
+            return ""
+        # replace invalid braces and quotes
+        string = string.replace("{", "[").replace("}", "]").replace("\"", "\'")
+        return string.strip()
+    call_command('create_subject_areas_dict')
     SubjectArea = apps.get_model('hs_dictionary.SubjectArea')
     UserProfile = apps.get_model('theme.UserProfile')
     # Attempt to match existing SAs from profiles
+    errors = []
     profiles_with_sa = UserProfile.objects \
         .exclude(subject_areas__isnull=True) \
         .exclude(subject_areas='')
@@ -21,8 +34,16 @@ def migrate_csv_subject_areas(apps, schema_editor):
         print(f'Searching user #{profile.pk} which has subject areas: {profile.subject_areas}')
         new_subj_areas = []
         for subject in old_subject_areas:
+            if subject == '':
+                # There is a trailing comma that we need to remove
+                continue
+            stripped_subject = strip_for_dict(subject)
+            if stripped_subject == '':
+                # The subject contained only invalid chars
+                print(f"- Unmatched subject area '{subject}' contains invalid chars that will be stripped")
+                continue
             print(f"Searching for a match with '{subject}'")
-            match = [sa for sa in subject_area_objects if sa.name.lower() == subject.strip().lower()]
+            match = [sa for sa in subject_area_objects if sa.name.lower() == stripped_subject.lower()]
             if match:
                 new_subj_areas.append(match[0].name)
                 if match[0].name == subject:
@@ -30,24 +51,33 @@ def migrate_csv_subject_areas(apps, schema_editor):
                 else:
                     print(f'- Near match with pre-existing subject area: {subject}')
             else:
-                if subject.strip() == subject:
+                if stripped_subject == subject:
                     print(f"- Unmatched subject area '{subject}' will remain unaltered")
                     new_subj_areas.append(subject)
                 else:
-                    print(f"- Unmatched subject area '{subject}' contains whitespace that will be stripped")
-                    new_subj_areas.append(subject.strip())
+                    print(f"- Unmatched subject area '{subject}' contains invalid chars that will be stripped")
+                    new_subj_areas.append(stripped_subject)
 
         sas = ','.join(new_subj_areas)
-        print(f'Updating {profile} from {profile.subject_areas} subject_areas to {{{sas}}}')
+        message = f'Updating {profile} from {profile.subject_areas} subject_areas to {{{sas}}}'
+        print(message)
         profile.subject_areas = f'{{{sas}}}'
+        try:
+            profile.save()
+        except DataError as e:
+            errors.append(f'Error saving profile: {e}' + message)
+
+    profiles_without_sa = UserProfile.objects.filter(subject_areas='')
+    for profile in profiles_without_sa:
+        print(f'Updating {profile} from "" to {{}}')
+        profile.subject_areas = '{}'
         profile.save()
 
-        profiles_without_sa = UserProfile.objects \
-            .filter(subject_areas='')
-        for profile in profiles_without_sa:
-            print(f'Updating {profile} from "" to {{}}')
-            profile.subject_areas = '{}'
-            profile.save()
+    print("Done updating Subject Areas.")
+    if errors:
+        print("Errors during update:")
+        for error in errors:
+            print(error)
 
 
 class Migration(migrations.Migration):


Test patch: