Repository: DDMAL/CantusDB

Issue:
strange user assignments?
As I was looking at the sources in #999 I noticed some VERY suspicious user-source relationships. I happen to know that the sources in Helsinki are being worked on by users 251662, 251663 and 251678--who I don't see at all in the recently modified sources list. Instead I see 251612 and 251613, who I didn't think had been logged in in some time, and aren't to my knowledge involved in that project, and all the Finnish sources seem to have been "last modified by" @jacobdgm. 

I find this odd...




Discussion:
Can you list the IDs of a few of these sources?
714693, 714807, 711232, 711225, 711223, 711204.

Now that I'm looking at the list I'm also a little suspicious of a number of other sources (713041, 713037, 713033) that aren't created by who I thought created them...



What I notice is that the affected users are all pretty recent accounts, and their replacements are mostly ones with fake emails attached.  
So for example source 714807 was made by user 251612 on OldCantus; the source is still attached to the userID 251612, but the actual user associated with that user ID is somebody else (one of the fake accounts), while the actual user of that OldCantus ID is now userID 251662. 
Strangely I have not heard any complaints from these users about not being able to access their sources, but maybe they just haven't tried yet...
huh - my guess as to what happened:
- a year ago, we synced users between OldCantus and NewCantus
- then, we merged users and indexers, creating a bunch of new dummy users
- new user accounts were created on OldCantus, and then sources were assigned to these new user accounts. The IDs for these accounts sometimes collided with IDs for indexer/dummy users on CantusDB
- we synced data from OldCantus to NewCantus. When we updated sources assigned to the new users on OldCantus, NewCantus checked to see whether a user account matching the user ID existed in NewCantus. If it did (i.e. if there was a dummy indexer/user with a matching user ID in NewCantus), we assigned that user to the source.

It might make most sense to sort this out manually. Unless I've overlooked something, we can go through all the users with @fakeemail.com accounts, and check to make sure that they don't have any "Sources Assigned to User". If they do, we check with OldCantus to see which user account they _should_ be associated with, add the sources to the proper user account, and remove them from the dummy account.
@annamorphism, does this sound like the right approach? Would it make sense for you to do this (since you have a better sense for who should belong to what source than I do)? I'd also be happy to do it - let me know.
I also have a pretty good idea of who is a new-ish user added on OldCantus. So yes, I can go through this. 
Wait--the problem is actually a little different. The fakeemail users don't have sources assigned to them as such, but their names show up as "created by"--so the OldCantus userID associated with that source got pulled into NewCantus and then "translated" into the name, as it were, instead of first getting mapped to the newCantus ID and then turned into a name. The source assignments, though, seem to have moved along with the users. On the other hand, affected sources have both their old and new users in the assigned editor box, with the same name: 
<img width="807" alt="image" src="https://github.com/DDMAL/CantusDB/assets/67451875/dee53b07-ffdb-4f3d-a953-13c9ce1daf42">
I'm assuming one of these is the OldCantusID of that user and one is the NewCantus one, but I don't know which one is which...
Anyway, I'm not sure I know how to fix what displays for "created by" through the admin menu, but I could make a list of affected sources/users. It seems like this problem doesn't affect access, which is good in the short term, but does mean the sources are incorrectly attributed, which is bad in the long term. 

And today I got my first email wondering why some person nobody knew was listed as a "contributor" on their source.  I think there are 17 sources affected, but I don't know how to change the "created by" information to be associated with the new cantus user and not the old one. 
> I'm assuming one of these is the OldCantusID of that user and one is the NewCantus one, but I don't know which one is which...

When I was manually importing recently-created users, I came across duplicated names like this, and it seems to be a bug - if you go to change a user's name, you can only find one of them in the Users app in the Admin area. And then if you change the name of one of them, _both_ instances are changed on the source page. So I'm pretty sure there is just one "Hilkka-Liisa Vuori", I'm just not sure why it's appearing twice.

(maybe we're taking the queryset of users, and the queryset of users who are indexers, and concatenating them without checking for duplicates? Let me look into this...)
Just following up on this - the duplicated names in the admin interface had to do with single user accounts belonging to multiple permission groups. But the issue at hand has to do with users being synced at one time while user-source assignments were synced significantly later. @annamorphism, do you have all the information you need to sort this out? Is there anything else I can do to help with this?
@jacobdgm no I 'm not sure we do. The duplicate names turned out to be a red herring (and spun off into the now-solved #1021). I can't change who shows up as a "contributor" to source 714693, and the fake user who shows up has no sources assigned to her I can modify either. The problem is somewhere in the pipeline that led to the created-by information, and I am not sure I can fix it. 
@annamorphism and I just talked about this - @annamorphism is going to create a csv (or similar) with two columns: one with source ID, and one with the User ID of the NewCantus user who authored the source. I'll write a script to go through the sources and assign them to these proper users.
I have made two sheets relevant to this problem. https://docs.google.com/spreadsheets/d/1DtUbkPBmrxYP3om-3ZmHNvsnfmBEbGR6dLrap16rZRE/edit?usp=sharing
One is a list of affected sources with the creatorID they *should* be associated with.
I feel fairly confident these 17 sources are the only affected ones, so if the "created by" information on these is changed we will be all set. 
The second sheet  is a list of users whose IDs were changed from OldCantus to NewCantus (which created the list of sources, and which may be helpful for other information moved from one place to the other e.g. chant creator information.) 
Method:
In principle anybody with an OldCantus UserID above 251599 was affected (which is the lowest ID assigned to a fake email) but for whatever reason 251599 through 251609 were not assigned, as were, apparently, 251628, 251629, 251634, 251635, 251636, and 251637. (I have included these in the sheet in strikethrough just in case I have simply overlooked the reason for these apparently unassigned IDs; if they are indeed unassigned, we can delete these rows.)  A few other IDs in this range were assigned for testing purposes to users who never got a NewCantus ID (or in one case, already had a different ID); these also can be deleted from the list. 
This leaves 27 affected users, whose NewCantus ID I looked up.  
I then looked up all the sources created since the last update, and included any that were created by affected users on the list of sources to change.




Resolving patch:
diff --git a/django/cantusdb_project/main_app/management/commands/remap_user_ids.py b/django/cantusdb_project/main_app/management/commands/remap_user_ids.py
new file mode 100644
--- /dev/null
+++ b/django/cantusdb_project/main_app/management/commands/remap_user_ids.py
@@ -0,0 +1,132 @@
+from main_app.models import Source, Chant
+from django.contrib.auth import get_user_model
+from django.core.management.base import BaseCommand
+from sys import stdout
+from django.db.models.query import QuerySet
+from typing import Optional
+
+User = get_user_model()
+
+USER_ID_MAPPING = {
+    # Fake user accounts with sequential numbering were created on NewCantus
+    # for OldCantus Indexers. In the time since user accounts were
+    # programmatically synced, new user accounts were created on OldCantus,
+    # which duplicated these IDs. Then, we manually created new user accounts
+    # on NewCantus for these newer users, with new IDs that don't match those
+    # in OldCantus.
+    #
+    # In this dictionary:
+    # - Keys represent the IDs of users recently created on OldCantus, which collide
+    #   with those of NewCantus Indexers
+    # - Values represent the IDs of manually-created users in NewCantus.
+    251610: 251660,
+    251611: 251661,
+    251612: 251662,
+    251613: 251663,
+    251614: 251664,
+    251616: 251665,
+    251617: 251666,
+    251618: 251667,
+    251619: 251668,
+    251620: 251669,
+    251621: 251670,
+    251622: 251671,
+    251623: 251672,
+    251624: 251673,
+    251625: 251674,
+    251626: 251657,
+    251627: 251675,
+    251630: 251676,
+    251632: 251678,
+    251633: 251679,
+    251638: 251656,
+    251639: 251680,
+    251640: 251681,
+    251641: 251682,
+    251642: 251683,
+    251643: 251684,
+    251645: 251685,
+}
+
+
+def reassign_sources() -> None:
+    CHUNK_SIZE = 1_000
+    sources: QuerySet[Source] = Source.objects.all()
+    sources_count: int = sources.count()
+    start_index: int = 0
+    while start_index <= sources_count:
+        stdout.write(f"processing chunk with {start_index=}\n")
+        chunk: QuerySet[Source] = sources[start_index : start_index + CHUNK_SIZE]
+        for source in chunk:
+            old_creator: Optional[User] = source.created_by
+
+            updated_id: Optional[int] = None
+            try:
+                updated_id: int = USER_ID_MAPPING[old_creator.id]
+            except (
+                KeyError,  # old_creator.id not in USER_ID_MAPPING
+                AttributeError,  # old_creator is None
+            ):
+                pass
+
+            if updated_id is None:
+                # user ID doesn't need to be remapped
+                continue
+
+            updated_creator: Optional[User] = None
+            try:
+                updated_creator = User.objects.get(id=updated_id)
+            except (
+                User.DoesNotExist,
+                AttributeError,
+            ):
+                pass
+
+            source.created_by = updated_creator
+            source.save()
+        start_index += CHUNK_SIZE
+
+
+def reassign_chants() -> None:
+    CHUNK_SIZE = 1_000
+    chants: QuerySet[Chant] = Chant.objects.all()
+    chants_count: int = chants.count()
+    start_index: int = 0
+    while start_index <= chants_count:
+        stdout.write(f"processing chunk with {start_index=}\n")
+        chunk: QuerySet[Chant] = chants[start_index : start_index + CHUNK_SIZE]
+        for chant in chunk:
+            old_creator: Optional[User] = chant.created_by
+
+            updated_id: Optional[int] = None
+            try:
+                updated_id: int = USER_ID_MAPPING[old_creator.id]
+            except (
+                KeyError,  # old_creator.id not in USER_ID_MAPPING
+                AttributeError,  # old_creator is None
+            ):
+                pass
+
+            if updated_id is None:
+                # user ID doesn't need to be remapped
+                continue
+
+            updated_creator: Optional[User] = None
+            try:
+                updated_creator = User.objects.get(id=updated_id)
+            except User.DoesNotExist:
+                pass
+
+            chant.created_by = updated_creator
+            chant.save()
+        start_index += CHUNK_SIZE
+
+
+class Command(BaseCommand):
+    def handle(self, *args, **kwargs) -> None:
+        stdout.write("\n\n==== Reassigning Sources ====\n")
+        reassign_sources()
+        stdout.write("\n== All sources successfully remapped! ==\n")
+        stdout.write("\n\n==== Reassigning Chants ====\n")
+        reassign_chants()
+        stdout.write("\n== All chants successfully remapped! ==\n")


Test patch: