Repository: conan-io/conan

Issue:
Performance of graph_builder got worse since 1.13.2
In our company we have decided to upgrade to the newest version of Conan (1.15.1). What was a surprise for me was that conan install took way more time than before. So I did some tracing with debug and found out that graph_builder is running 550 seconds instead of previous 8. This happends on libraries which have large count of dependencies (full_requires item count in conan_info.txt > 60) and some of the dependencies have also large count of dependencies. The very dependent libraries are probably making problem there.

From log I can see that in 1.13.1 took graph_builder less than second to run for the very dependent dependency (its full_requires item count = 64), in 1.13.2 it takes 78 seconds.

I was able to detect version where this started and it is 1.13.2. With 1.13.1 I do not have such problem.

Unfortunately I cannot provide you a packages for reproducing because they are company asset.

Is it known bug or is this happening just to me?


Discussion:
Hi @jakuborava 

Thanks very much for reporting.
Yes, we have changed the internal dependency resolution algorithm, because it had some bug that was potentially dangerous. We have some benchmark we use to monitor time, that uses graphs of 50-250 nodes, and we didn't detect any substantial change, but it is possible that some of the changes are affecting you. Some quick questions:

- That ``conan install`` is fetching the packages from the server, or it is resolving everything to the local cache.
- What is the time if using only packages installed in the local cache. I am trying to rule out that many more http api calls are being made for some reason.
- What is the topology of your graph, it is very connected, like tons of packages depending on each other?
- Are you using version-ranges, or fixed dependencies?
- How many nodes total has your graph?
- Heavy usage of build_requires or private deps?

I am running some benchmarks, will update soon.
I am experimenting with graphs with 50 nodes, and getting times <1 sec for a install with pre-compiled cached binaries, with different simple topologies.
Hi @memsharded,

thanks for a quick reply! To answer your questions:

> That conan install is fetching the packages from the server, or it is resolving everything to the local cache.

Yes this use case was done with empty cache, so everything was resolved from remote repository.

> What is the time if using only packages installed in the local cache. I am trying to rule out that many more http api calls are being made for some reason.

The time is the same even if working with local cache.

> Are you using version-ranges, or fixed dependencies?

Fixed dependencies are used.

> Heavy usage of build_requires or private deps?

No. Every package has two basically same build_requires and no private deps are used.

I will provide graph info as soon as possible.
I can provide graph screenshot:
![graph](https://user-images.githubusercontent.com/9501966/58565126-797f4080-822e-11e9-9ff9-5fc3d845db87.png)

There is something around 70 nodes and as you can see density of the graph is huge.

I have tried to add a couple of build_requires to each node of the graph, also adding more density, still can't see any significant extra cost, I keep trying things.
If it will help I could try provide you obfuscated conanfiles tomorrow, then you will have a real use case that you can also debug.
Oh, that would be really useful, please do! And remember that even if obfuscated, you prefer to send them privately, you can write to info@conan.io. Many thanks for the help!
Zip has been sent over email. I hope it will be useful.

Resolving patch:
diff --git a/conans/client/graph/graph.py b/conans/client/graph/graph.py
--- a/conans/client/graph/graph.py
+++ b/conans/client/graph/graph.py
@@ -44,12 +44,6 @@ def __init__(self, ref, conanfile, recipe=None):
         self.inverse_closure = set()  # set of nodes that have this one in their public
         self.ancestors = None  # set{ref.name}
 
-    def update_ancestors(self, ancestors):
-        # When a diamond is closed, it is necessary to update all upstream ancestors, recursively
-        self.ancestors.update(ancestors)
-        for n in self.neighbors():
-            n.update_ancestors(ancestors)
-
     @property
     def package_id(self):
         return self._package_id
diff --git a/conans/client/graph/graph_builder.py b/conans/client/graph/graph_builder.py
--- a/conans/client/graph/graph_builder.py
+++ b/conans/client/graph/graph_builder.py
@@ -191,7 +191,9 @@ def _handle_require(self, name, node, require, dep_graph, check_updates, update,
                                      % (node.ref, require.ref, previous.ref))
 
             # Add current ancestors to the previous node
-            previous.update_ancestors(node.ancestors.union([node.name]))
+            union = node.ancestors.union([node.name])
+            for prev_node in previous.public_closure.values():
+                prev_node.ancestors.update(union)
             if previous.private and not require.private:
                 previous.make_public()
 


Test patch: