Repository: godotengine/godot (language: cpp)

Issue:
Godot choosing the wrong scenes when duplicate UUIDs exist
- *Production edit: Related to https://github.com/godotengine/godot/issues/87580.*

### Tested versions

- Reproducible in 4.4-dev (ce8a837aab2568f9cdc41b4b410c478b0cd711fc)
- Reproducible in 4.3 stable

### System information

Windows 10

### Issue description

Mentioned this on `core` on Rocketchat but the plot thickened so I am raising this as an issue just in case we can find good ways to deal with this.

The issue I reported on Rocketchat was somewhat self inflicted, when starting new project I often copy stuff I need from other projects, but as some of these were also copied into a plugin that I started using, weird clashes ensued and I spend a fair bit of time cleaning up the mess.

But my problem is actually larger then this. I'm working on a new prototype for our XR Tools plugin (aptly named XR Tools 2, though that is going to bite me in the A$$ so that is unlikely to be final). This new version is structurally different hence starting from scratch, but I copy across many of the original XR tools scenes where I can reuse things, or use the old implementation as a base for the new implementation. This copying off course happens on the file system, NOT in the IDE.
The two plugins thus share MANY UUIDs

However, when migrating an existing project from the original XR Tools to XR Tools 2, you will likely start by adding the new version of the plugin to the project and slowly moving things over, it is not a drop in replacement. At this point our problem starts happening.

Godot will load all the scenes for the original XR Tools, then it will load all the scenes for XR Tools 2 however when it encounters referenced UUIDs that it has already loaded, instead of verifying if this UUID is correct, it will CHANGE XR TOOLS 2 TO USE XR TOOLS 1s SUBSCENES. When the subscene then gets loaded it detects that UUID is already in use and gives that scene a new UUID completely severing any ties that this subscene was the actually one used. This breaks the entire plugin.

It gets worse from there, because even if I fix up the subscenes in this project, the UUIDs are now different from the upstream project so I will have a continued fight over the lifecycle of this project every time I pull down an updated version of the plugin that restores the original UUIDs.

I don't think this scenario will be unique, when a major rebuild of a plugin is the only way to get rid of technical debt, it is likely things will be copied from old to new.

While I don't think the problem can be entirely fixed, duplicating files on the file system will duplicate UUIDs and while Godot tries to recover from this, it may not recover from this properly in all scenarios, I do think think there is one improvement we can contemplate here.

As part of marking a folder as a plugin, plugins should maintain their own UUID so that duplicate UUIDs can exist between plugins.
This means that when a class in a plugin is referenced from outside of that plugin, it should be prefixed somehow. Though I wonder how we can deal with dependencies between plugins.  

### Steps to reproduce

Easiest way to reproduce this I guess is to start with a project that has a plugin with scenes that include subscenes, and duplicate that plugin.

### Minimal reproduction project (MRP)

N/A


Discussion / hints:
It's not clear if the problem comes from scenes or scripts. If it comes from scenes, could a tool to generate a new UUID exist? It could solve the issue if you're a plugin maker that copy files from another plugin.
Does this also happen when you drag the files onto the godot filedock instead of just copying them on disk? I think if you drop them on the dock it should regenerate the UUIDs but not sure.
> It's not clear if the problem comes from scenes or scripts. If it comes from scenes, could a tool to generate a new UUID exist? It could solve the issue if you're a plugin maker that copy files from another plugin.

Really any file that gets a UUID and is referenced by it, but the main problems I was dealing with today were with scenes used as subscenes.

The problem is, how does Godot know to use new UUIDs? When building a new version of an existing plugin, or when re-using scenes from another project for a new project, Godot just sees new scenes with a UUID that it's happy to use.

The problems don't start until you get into a situation when you combine things, such as adding both old and new version of the plugin to the same project, which is a likely scenario when migrating an existing project from using an old version of the plugin to the new version of the plugin (again in the complete rewrite scenario, not normal version upgrades where you would just overwrite the plugin and deal with the changes).

Changing UUIDs here is disastrous through the continued lifecycle of that project. At some point when migration is done you will delete the old plugin, and now you have the new plugin in this project with all new UUIDs. When you go and update that plugin to a newer minor version where you overwrite stuff, you go back to the old UUIDs because those still exists upstream. While likely Godot will recover from this, it still opens up the possibility for the wrong scenes to be referenced and breaking your application.

Also when changes are made to the plugin which are then up-streamed, you now get a mix of old and new UUIDs as upstream we're still using the original UUIDs.
This is actually a major concern in XR Tools 2 where it's now optionally possible to submodule the plugin, which makes it much easier to develop new features (I love git submodule, soo powerful when used correctly), though we ran afoul of this very often with the current XR Tools as people copy the changes they made in their project, back to a copy of their fork of XR Tools to push in those changes, with now mismatches of UUIDs as Godot opening their project assigned new UUIDs to scenes. 
I think the root of the problem is that the uid system interacts poorly with adding externally created resources into a project (including scenes made in another project, or plugin):

Such resources have their own uids written in the `.tres`, but they are not parsed by the uid cache until the resource is loaded. This results in path fallback (and an annoying error). If the uid is already registered it loads the wrong resource.

I think the first step to solving this would be to make it so that every time a new uid having file is added to the filesystem, the uid is either registered if not there or changed if there. This wouldn't fix the vcs interaction, which is unfortunate, but should stop things from breaking.
About making it work well with vcs:

I really think scenes, etc. made for downloading should not contain or reference uids at all. Files in a repository are going to stay with the same paths, so path references will stay valid.

This would mean that, when creating an addon, there would need to be an option to strip uids, and when downloading an addon, they will be created for the specific project. This way in whatever project context an addon is used, it can be modified locally, have it's uids stripped and will generate a clean diff.
@Jordyfel 

The problem is that this still means new UIDs are created when importing a plugin for the first time, that happens to have overlapping UIDs with existing classes, and thus the UIDs of the plugin changing. When you update that plugin to a newer version, the old UIDs return back into your project and things again risk to be broken.
One solution I tried this week was to change the UIDs of the new version of my plugin manually to UIDs that are prefixed with something unique.

Problem is, Godot doesn't like my new UIDs and sporadically changes them to newly generated UIDs. I couldn't confirm whether it does this because it finds my custom chosen UIDs invalid, or whether there is a bug in the save logic of Godot which will change UIDs when it shouldn't.

Discussed this with @KoBeWi and he thinks its probably because I broke some formatting rules around UIDs.

Just for reference, #92972 is a nice enhancement but won't help me here because the project I use to develop the new version of the plugin won't trigger any duplicate UID warning. It's only when the plugin is added to an existing project that uses the old plugin because someone wants to migrate over to the new version, that issues start to happen.
Still a nice enhancement.
^ cc @Chaosus @Calinou 
> One solution I tried this week was to change the UIDs of the new version of my plugin manually to UIDs that are prefixed with something unique.

Why would you change your UIDs? The same file should keep the same UID, this way you won't have problems.
EDIT:
Nevermind, I've read the OP.

> duplicating files on the file system will duplicate UUIDs

No, they will have new UIDs assigned.

---

If you plan on using both old and new plugin files, the easiest way would be duplicate the whole old plugin folder in Godot editor, so you will get a copy with new UIDs.
@KoBeWi 

> No, they will have new UIDs assigned.

This has two problems.
1) if my original class UID gets changed, and the new class retains the duplicate UID, all my existing code now points to the WRONG class
2) if I'm an author/contributor for the plugin I'm bringing in, these changed UIDs will be uploaded back into the project and cause further problems.

And again, I'm not using them both, in my use case I am developing the new plugin separately and most people will use it as is. It is the people that are using the old plugin and want to migrate that have a problem. They will have existing projects, now the new plugin is brought in, IF Godot decides to renumber UIDs in the old plugin, things break (which I've already proven is exactly what happens). Especially because the new plugin is setup that it can be submoduled into the project, so you would normally do this outside of Godot.

But I was discussing this further with others. Here is another scenario that is very likely to happen. I have a sample project on `godot-demo-projects` showing how to use compositor effects. We even reference this in our documentation. This contains a lot of boilerplate code. 

Someone who wants to create a compositor effect is likely going to copy my project and use that as a base. Their compositor effect will thus have the same UID as the compositor effect in the demo.

If the effect is cool, they are going to share it with other users.

The more people create compositor effects, the more effects will be obtainable through the asset library that all share the same UIDs because they all have the same ancestor implementation.

So if I am working on a project for months that includes such a compositor effect, and I've referenced it in many places, and I now see someone elses effect that I'd like to also include in my project. I run a realistic risk of pulling in code with a duplicate UID.
IF in this scenario Godot decides to renumber the existing effect, instead of the new effect.. TROUBLE

But worse yet, if this person UPDATES their effect, and I pull in the new version, and I thus pull in the OLD DUPLICATE UID, I could be in even more trouble, especially if this person moved files around in their asset and Godot has no idea how to decipher the mess that has been created.

We have to have a way to namespace UIDs within plugins so that we can work with multiple assets having duplicate UIDs because they have the same ancester code.
What is the purpose of that UUID? (eg can be it created for exported builds only or something?) I was copying files around in a gamejam, and things broke because of this. This is not even an edge case, it's extremely easy to trigger it.

Fix patch:
diff --git a/editor/editor_file_system.cpp b/editor/editor_file_system.cpp
index 89cba03d4ebf..478a19d6412e 100644
--- a/editor/editor_file_system.cpp
+++ b/editor/editor_file_system.cpp
@@ -843,11 +843,26 @@ bool EditorFileSystem::_update_scan_actions() {
 
 				fs_changed = true;
 
+				const String new_file_path = ia.dir->get_file_path(idx);
+				const ResourceUID::ID existing_id = ResourceLoader::get_resource_uid(new_file_path);
+				if (existing_id != ResourceUID::INVALID_ID) {
+					const String old_path = ResourceUID::get_singleton()->get_id_path(existing_id);
+					if (old_path != new_file_path && FileAccess::exists(old_path)) {
+						const ResourceUID::ID new_id = ResourceUID::get_singleton()->create_id();
+						ResourceUID::get_singleton()->add_id(new_id, new_file_path);
+						ResourceSaver::set_uid(new_file_path, new_id);
+						WARN_PRINT(vformat("Duplicate UID detected for Resource at \"%s\".\nOld Resource path: \"%s\". The new file UID was changed automatically.", new_file_path, old_path));
+					} else {
+						// Re-assign the UID to file, just in case it was pulled from cache.
+						ResourceSaver::set_uid(new_file_path, existing_id);
+					}
+				}
+
 				if (ClassDB::is_parent_class(ia.new_file->type, SNAME("Script"))) {
-					_queue_update_script_class(ia.dir->get_file_path(idx), ia.new_file->type, ia.new_file->script_class_name, ia.new_file->script_class_extends, ia.new_file->script_class_icon_path);
+					_queue_update_script_class(new_file_path, ia.new_file->type, ia.new_file->script_class_name, ia.new_file->script_class_extends, ia.new_file->script_class_icon_path);
 				}
 				if (ia.new_file->type == SNAME("PackedScene")) {
-					_queue_update_scene_groups(ia.dir->get_file_path(idx));
+					_queue_update_scene_groups(new_file_path);
 				}
 
 			} break;


Test patch: