Repository: conan-io/conan

Issue:
Occassional random failure of package uploads on CI system
My CI is set up to build Conan packages as soon as the new feature is merged into the stable branch. The CI system must build Conan packages for all platforms we support, namely Android (all ABIs), iOS (all architectures), Windows, MacOS and Linux (both with clang and gcc). We are using Jenkins as our CI system and step that performs the build is done in parallel on multiple Jenkins slaves. Since we do not want to upload any packages until we are completely certain that the building of packages has succeeded on all platforms, each slave does not upload built packages directly to the Artifactory. Instead, each slave performs `conan upload * -r myRemote --all --skip-upload --confirm` so that it prepares `.tgz` file, and then stashes it with using Jenkins' `stash` command (you can find more info [here](https://github.com/conan-io/conan/issues/3452#issuecomment-417922420) why we are doing that this way).

Finally, after all nodes have completed the building of packages, a single node then unstashes all the artifacts and uploads them to the Artifactory. However, with Conan v1.13 and later, the created package also contains file `metadata.json`, which contains some meta information about built binaries. However, when multiple nodes have been building different versions of packages on different machines, the built packages contain only the partial metadata.json file. When the "upload node" unstashes all built binaries, it continuously overwrites the `metadata.json` with each stash. This then makes it impossible for conan to upload all the binaries because the final `metadata.json` does not contain references to all package IDs available and it fails.

One possible solution for that would be to manually merge the metadatas using Jenkins Groovy scripting, but since I found that rather complex, I implemented a different approach - spawn new N parallel jobs (using Jenkins' `parallel` command), each on a separate node (just like when building), where each node will upload only a single stash. Thus, the "merging" of metadata.json actually happens on the Artifactory server and the upload is actually faster, as Conan still does not support concurrent uploading of packages (see issue #3452).

However, occassionally, at completely random, one or more parallel upload "jobs" fail with following stack trace:

```
[b] $ sh -c "conan upload \* -r 090a7942-cd0f-45af-9347-465cbbe94a6e --confirm --all --no-overwrite "
MicroblinkConanFile/3.0.2@microblink/stable: Not found in local cache, looking in remotes...
MicroblinkConanFile/3.0.2@microblink/stable: Trying with '090a7942-cd0f-45af-9347-465cbbe94a6e'...
Downloading conanmanifest.txt

Downloading conanfile.py

MicroblinkConanFile/3.0.2@microblink/stable: Downloaded recipe revision 0
Uploading to remote '090a7942-cd0f-45af-9347-465cbbe94a6e':
Uploading CoreUtils/2.2.0@microblink/master to remote '090a7942-cd0f-45af-9347-465cbbe94a6e'
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/conans/client/command.py", line 1579, in run
    method(args[0][1:])
  File "/usr/lib/python3.7/site-packages/conans/client/command.py", line 1182, in upload
    retry_wait=args.retry_wait, integrity_check=args.check)
  File "/usr/lib/python3.7/site-packages/conans/client/conan_api.py", line 93, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/conans/client/conan_api.py", line 868, in upload
    retry_wait, integrity_check, policy, remote_name, query=query)
  File "/usr/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 88, in upload
    integrity_check, policy, remote, upload_recorder)
  File "/usr/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 190, in _upload_ref
    self._upload_recipe(ref, conanfile, retry, retry_wait, policy, recipe_remote)
  File "/usr/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 230, in _upload_recipe
    remote, remote_manifest)
  File "/usr/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 344, in _recipe_files_to_upload
    if remote_manifest == local_manifest:
  File "/usr/lib/python3.7/site-packages/conans/model/manifest.py", line 132, in __eq__
    return self.file_sums == other.file_sums
AttributeError: 'NoneType' object has no attribute 'file_sums'

ERROR: 'NoneType' object has no attribute 'file_sums'
```

I've currently worked around that by retrying the upload several times until it succeeds, but this is a hacky and dirty solution I would like to remove from my Jenkins pipeline script.

Unfortunately, I don't have a deterministic way of reproducing the above issue, as it never happens on my development machine (which never uploads anything in parallel), so I hope that above stack trace will help you somehow track down the part of Conan's code that may be troubling.

Our CI servers use conan client 1.14.3 and the Artifactory being used is v6.9.0 Community Edition for C++.


Discussion:
Thanks for reporting. I've seen this error randomly also with Bintray. 
I'll try to investigate.
About the issues with the "stash", I think you should open a new issue because it is a totally different issue. That approach is indeed very problematic for the `metadata.json` maybe we can discuss or figure out a better one. (Related to https://github.com/conan-io/conan/issues/3073)
This issue about the `ERROR: 'NoneType' object has no attribute 'file_sums'` might be a duplicate of https://github.com/conan-io/conan/issues/4953
Thank you for answering to the report at such a quick notice.

> About the issues with the "stash", I think you should open a new issue because it is a totally different issue. That approach is indeed very problematic for the metadata.json maybe we can discuss or figure out a better one. (Related to #3073)

I wouldn't call this an issue since this is not something that is part of Conan's design. As Conan does not support uploading of rogue `.tgz` files (something that was requested in #3073), I am OK with that that I need to merge `metadata.json` manually, or even better, perform the upload of packages in parallel from multiple machines. Since support for concurrent upload is still not here (issue #3452), I actually get a faster upload by making it from different Jenkins nodes - the available network bandwidth gets used more efficiently.

However, this leads to some race conditions that you mentioned in [issue 4953 (comment)](https://github.com/conan-io/conan/issues/4953#issuecomment-485395006), since my workflow is highly parallel (see the screenshot from my Jenkins job below - the first attempt failed with the above error, but the second succeeded).
![screenshot](https://user-images.githubusercontent.com/158844/56503799-d17fa480-6516-11e9-99e3-04a7cf26c4b0.png)

Therefore, I think this probably is a duplicate of #4953.
 

Resolving patch:
diff --git a/conans/client/cmd/uploader.py b/conans/client/cmd/uploader.py
--- a/conans/client/cmd/uploader.py
+++ b/conans/client/cmd/uploader.py
@@ -208,13 +208,7 @@ def _upload_ref(self, conanfile, ref, prefs, retry, retry_wait, integrity_check,
                                    remote=recipe_remote)
 
     def _upload_recipe(self, ref, conanfile, retry, retry_wait, policy, remote):
-        if policy != UPLOAD_POLICY_FORCE:
-            remote_manifest = self._check_recipe_date(ref, remote)
-        else:
-            remote_manifest = None
-
         current_remote = self._registry.refs.get(ref)
-
         if remote != current_remote:
             complete_recipe_sources(self._remote_manager, self._cache, conanfile, ref)
 
@@ -224,10 +218,18 @@ def _upload_recipe(self, ref, conanfile, retry, retry_wait, policy, remote):
 
         t1 = time.time()
         the_files = self._compress_recipe_files(ref)
+        local_manifest = FileTreeManifest.loads(load(the_files["conanmanifest.txt"]))
+
+        remote_manifest = None
+        if policy != UPLOAD_POLICY_FORCE:
+            remote_manifest = self._check_recipe_date(ref, remote, local_manifest)
         if policy == UPLOAD_POLICY_SKIP:
             return ref
+
         files_to_upload, deleted = self._recipe_files_to_upload(ref, policy, the_files,
-                                                                remote, remote_manifest)
+                                                                remote, remote_manifest,
+                                                                local_manifest)
+
         if files_to_upload or deleted:
             self._remote_manager.upload_recipe(ref, files_to_upload, deleted,
                                                remote, retry, retry_wait)
@@ -333,13 +335,27 @@ def _compress_package_files(self, pref, integrity_check):
         the_files = _compress_package_files(files, symlinks, package_folder, self._user_io.out)
         return the_files
 
-    def _recipe_files_to_upload(self, ref, policy, the_files, remote, remote_manifest):
-        # Get the remote snapshot
+    def _recipe_files_to_upload(self, ref, policy, the_files, remote, remote_manifest,
+                                local_manifest):
         self._remote_manager.check_credentials(remote)
         remote_snapshot = self._remote_manager.get_recipe_snapshot(ref, remote)
+        files_to_upload = {filename.replace("\\", "/"): path
+                           for filename, path in the_files.items()}
+        if not remote_snapshot:
+            return files_to_upload, set()
 
-        if remote_snapshot and policy != UPLOAD_POLICY_FORCE:
-            local_manifest = FileTreeManifest.loads(load(the_files["conanmanifest.txt"]))
+        deleted = set(remote_snapshot).difference(the_files)
+        if policy != UPLOAD_POLICY_FORCE:
+            if remote_manifest is None:
+                # This is the weird scenario, we have a snapshot but don't have a manifest.
+                # Can be due to concurrency issues, so we can try retrieve it now
+                try:
+                    remote_manifest, _ = self._remote_manager.get_recipe_manifest(ref, remote)
+                except NotFoundException:
+                    # This is weird, the manifest still not there, better upload everything
+                    self._user_io.out.warn("The remote recipe doesn't have the 'conanmanifest.txt' "
+                                           "file and will be uploaded: '{}'".format(ref))
+                    return files_to_upload, deleted
 
             if remote_manifest == local_manifest:
                 return None, None
@@ -348,9 +364,6 @@ def _recipe_files_to_upload(self, ref, policy, the_files, remote, remote_manifes
                 raise ConanException("Local recipe is different from the remote recipe. "
                                      "Forbidden overwrite.")
 
-        files_to_upload = {filename.replace("\\", "/"): path
-                           for filename, path in the_files.items()}
-        deleted = set(remote_snapshot).difference(the_files)
         return files_to_upload, deleted
 
     def _package_files_to_upload(self, pref, policy, the_files, remote):
@@ -405,13 +418,12 @@ def _package_integrity_check(self, pref, files, package_folder):
             self._user_io.out.rewrite_line("Package integrity OK!")
         self._user_io.out.writeln("")
 
-    def _check_recipe_date(self, ref, remote):
+    def _check_recipe_date(self, ref, remote, local_manifest):
         try:
             remote_recipe_manifest, ref = self._remote_manager.get_recipe_manifest(ref, remote)
         except NotFoundException:
             return  # First time uploading this package
 
-        local_manifest = self._cache.package_layout(ref).recipe_manifest()
         if (remote_recipe_manifest != local_manifest and
                 remote_recipe_manifest.time > local_manifest.time):
             self._print_manifest_information(remote_recipe_manifest, local_manifest, ref, remote)


Test patch:
diff --git a/conans/test/functional/command/upload_test.py b/conans/test/functional/command/upload_test.py
--- a/conans/test/functional/command/upload_test.py
+++ b/conans/test/functional/command/upload_test.py
@@ -3,16 +3,20 @@
 import unittest
 from collections import OrderedDict
 
-from mock import mock
+from mock import mock, patch
+
 
 from conans.client.tools.env import environment_append
 from conans.errors import ConanException
 from conans.model.ref import ConanFileReference, PackageReference
 from conans.paths import EXPORT_SOURCES_TGZ_NAME, PACKAGE_TGZ_NAME
 from conans.test.utils.cpp_test_files import cpp_hello_conan_files
-from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient, TestServer
+from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient, TestServer,\
+    TurboTestClient
 from conans.util.files import gzopen_without_timestamps, is_dirty, save
 from conans.test.utils.conanfile import TestConanFile
+from conans.client.cmd.uploader import CmdUpload
+
 
 conanfile = """from conans import ConanFile
 class MyPkg(ConanFile):
@@ -362,7 +366,8 @@ def package(self):
         # *1
         client.run("upload Hello0/1.2.1@frodo/stable --all --no-overwrite",
                    assert_error=not client.cache.config.revisions_enabled)
-        if not client.cache.config.revisions_enabled:  # The --no-overwrite makes no sense with revisions
+        if not client.cache.config.revisions_enabled:
+            # The --no-overwrite makes no sense with revisions
             self.assertIn("Forbidden overwrite", client.out)
             self.assertNotIn("Uploading conan_package.tgz", client.out)
 
@@ -553,10 +558,8 @@ def upload_login_prompt_disabled_user_authenticated_test(self):
 
     def upload_key_error_test(self):
         files = cpp_hello_conan_files("Hello0", "1.2.1", build=False)
-        server1 = TestServer([("*/*@*/*", "*")], [("*/*@*/*", "*")],
-                                     users={"lasote": "mypass"})
-        server2 = TestServer([("*/*@*/*", "*")], [("*/*@*/*", "*")],
-                                     users={"lasote": "mypass"})
+        server1 = TestServer([("*/*@*/*", "*")], [("*/*@*/*", "*")], users={"lasote": "mypass"})
+        server2 = TestServer([("*/*@*/*", "*")], [("*/*@*/*", "*")], users={"lasote": "mypass"})
         servers = OrderedDict()
         servers["server1"] = server1
         servers["server2"] = server2
@@ -592,3 +595,42 @@ def upload_export_pkg_test(self):
         metadata = client.cache.package_layout(ref).load_metadata()
         self.assertIn(NO_SETTINGS_PACKAGE_ID, metadata.packages)
         self.assertTrue(metadata.packages[NO_SETTINGS_PACKAGE_ID].revision)
+
+    def test_no_remote_recipe_manifest(self):
+        # https://github.com/conan-io/conan/issues/4953
+        server = TestServer()
+        servers = OrderedDict([("default", server)])
+        client = TurboTestClient(servers=servers)
+        client2 = TurboTestClient(servers=servers)
+
+        ref = ConanFileReference.loads("lib/1.0@conan/testing")
+        client.create(ref)
+        complete_ref = client.upload_all(ref)
+        # Simulate a missing manifest, maybe because it hasn't been uploaded yet
+        export_folder = server.server_store.export(complete_ref)
+        os.unlink(os.path.join(export_folder, "conanmanifest.txt"))
+
+        # Upload same with client2
+        client2.create(ref)
+        client2.upload_all(ref)
+        self.assertIn("WARN: The remote recipe doesn't have the 'conanmanifest.txt' file "
+                      "and will be uploaded: 'lib/1.0@conan/testing'", client2.out)
+
+    def test_concurrent_upload(self):
+        # https://github.com/conan-io/conan/issues/4953
+        server = TestServer()
+        servers = OrderedDict([("default", server)])
+        client = TurboTestClient(servers=servers)
+        client2 = TurboTestClient(servers=servers)
+
+        ref = ConanFileReference.loads("lib/1.0@conan/testing")
+        client.create(ref)
+        client.upload_all(ref)
+        # The _check_recipe_date returns None, but later it will get the manifest ok
+        with patch.object(CmdUpload, "_check_recipe_date") as check_date:
+            check_date.return_value = None
+            # Upload same with client2
+            client2.create(ref)
+            client2.run("upload lib/1.0@conan/testing")
+            self.assertIn("Recipe is up to date, upload skipped", client2.out)
+            self.assertNotIn("WARN", client2.out)