Repository: conan-io/conan
Pull request #17708 (2025-02-05 12:01:52)

Fix 200% untar progress issue by avoiding a second read of file
Changelog: Fix: Improve untar performance
Docs: omit

Calling `TarFile.getmembers()` forces a complete IO read of the tar file.
The same happens when calling `TarFile.extractall(members=members)` for the first time.

This PR aims to fix the reported issue of having untar progress exceeding 100% occasionated by calling `seek(0)` twice, one in the first read querying all files and a second for the actual uncompression.

### Details:
Perform the decompression in a single IO loop by iterating over compressed files without actually reading them, filtering and uncompressing when needed one at a time.

### Benchmarks:

#### develop2 untar code
```
$ time conan source recipes/boost/all --version 1.86.0                                                                   
```
<details>

```
conanfile.py (boost/1.86.0): Calling source() in /Users/perseo/sources/conan-center-index/recipes/boost/all/src
conanfile.py (boost/1.86.0): Source ['https://archives.boost.io/release/1.86.0/source/boost_1_86_0.tar.bz2', 'https://sourceforge.net/projects/boost/files/boost/1.86.0/boost_1_86_0.tar.bz2'] retrieved from local download cache
conanfile.py (boost/1.86.0): Unzipping boost_1_86_0.tar.bz2 to /Users/perseo/sources/conan-center-index/recipes/boost/all/src
Uncompressing boost_1_86_0.tar.bz2: 5%
Uncompressing boost_1_86_0.tar.bz2: 10%
Uncompressing boost_1_86_0.tar.bz2: 22%
Uncompressing boost_1_86_0.tar.bz2: 30%
Uncompressing boost_1_86_0.tar.bz2: 42%
Uncompressing boost_1_86_0.tar.bz2: 54%
Uncompressing boost_1_86_0.tar.bz2: 62%
Uncompressing boost_1_86_0.tar.bz2: 71%
Uncompressing boost_1_86_0.tar.bz2: 79%
Uncompressing boost_1_86_0.tar.bz2: 89%
Uncompressing boost_1_86_0.tar.bz2: 95%
Uncompressing boost_1_86_0.tar.bz2: 100%
Uncompressing boost_1_86_0.tar.bz2: 103%
Uncompressing boost_1_86_0.tar.bz2: 105%
Uncompressing boost_1_86_0.tar.bz2: 106%
Uncompressing boost_1_86_0.tar.bz2: 108%
Uncompressing boost_1_86_0.tar.bz2: 112%
Uncompressing boost_1_86_0.tar.bz2: 121%
Uncompressing boost_1_86_0.tar.bz2: 124%
Uncompressing boost_1_86_0.tar.bz2: 127%
Uncompressing boost_1_86_0.tar.bz2: 130%
Uncompressing boost_1_86_0.tar.bz2: 139%
Uncompressing boost_1_86_0.tar.bz2: 146%
Uncompressing boost_1_86_0.tar.bz2: 152%
Uncompressing boost_1_86_0.tar.bz2: 156%
Uncompressing boost_1_86_0.tar.bz2: 161%
Uncompressing boost_1_86_0.tar.bz2: 165%
Uncompressing boost_1_86_0.tar.bz2: 170%
Uncompressing boost_1_86_0.tar.bz2: 172%
Uncompressing boost_1_86_0.tar.bz2: 176%
Uncompressing boost_1_86_0.tar.bz2: 179%
Uncompressing boost_1_86_0.tar.bz2: 184%
Uncompressing boost_1_86_0.tar.bz2: 189%
Uncompressing boost_1_86_0.tar.bz2: 193%
Uncompressing boost_1_86_0.tar.bz2: 195%
Uncompressing boost_1_86_0.tar.bz2: 196%
Uncompressing boost_1_86_0.tar.bz2: 199%
conanfile.py (boost/1.86.0): Apply patch (conan): Optional flag to specify iconv from either libc of libiconv
conan source recipes/boost/all    25,42s user 10,91s system 94% cpu 38,446 total
```
</details>

#### New changes
```
$  time conan source recipes/boost/all --version 1.86.0
```
<details>

```
conanfile.py (boost/1.86.0): Calling source() in /Users/perseo/sources/conan-center-index/recipes/boost/all/src
conanfile.py (boost/1.86.0): Source ['https://archives.boost.io/release/1.86.0/source/boost_1_86_0.tar.bz2', 'https://sourceforge.net/projects/boost/files/boost/1.86.0/boost_1_86_0.tar.bz2'] retrieved from local download cache
conanfile.py (boost/1.86.0): Unzipping boost_1_86_0.tar.bz2 to /Users/perseo/sources/conan-center-index/recipes/boost/all/src
Uncompressing boost_1_86_0.tar.bz2: 1%
Uncompressing boost_1_86_0.tar.bz2: 4%
Uncompressing boost_1_86_0.tar.bz2: 5%
Uncompressing boost_1_86_0.tar.bz2: 7%
Uncompressing boost_1_86_0.tar.bz2: 9%
Uncompressing boost_1_86_0.tar.bz2: 11%
Uncompressing boost_1_86_0.tar.bz2: 18%
Uncompressing boost_1_86_0.tar.bz2: 21%
Uncompressing boost_1_86_0.tar.bz2: 24%
Uncompressing boost_1_86_0.tar.bz2: 28%
Uncompressing boost_1_86_0.tar.bz2: 31%
Uncompressing boost_1_86_0.tar.bz2: 41%
Uncompressing boost_1_86_0.tar.bz2: 47%
Uncompressing boost_1_86_0.tar.bz2: 52%
Uncompressing boost_1_86_0.tar.bz2: 56%
Uncompressing boost_1_86_0.tar.bz2: 61%
Uncompressing boost_1_86_0.tar.bz2: 64%
Uncompressing boost_1_86_0.tar.bz2: 69%
Uncompressing boost_1_86_0.tar.bz2: 72%
Uncompressing boost_1_86_0.tar.bz2: 74%
Uncompressing boost_1_86_0.tar.bz2: 77%
Uncompressing boost_1_86_0.tar.bz2: 82%
Uncompressing boost_1_86_0.tar.bz2: 87%
Uncompressing boost_1_86_0.tar.bz2: 92%
Uncompressing boost_1_86_0.tar.bz2: 94%
Uncompressing boost_1_86_0.tar.bz2: 95%
Uncompressing boost_1_86_0.tar.bz2: 96%
conanfile.py (boost/1.86.0): Apply patch (conan): Optional flag to specify iconv from either libc of libiconv
conan source recipes/boost/all    15,30s user 10,32s system 89% cpu 28,726 total
```
</details>

The results show a 25% impact on uncompression efficiency

Closes https://github.com/conan-io/conan/issues/17697

----

- [x] Refer to the issue that supports this Pull Request.
- [x] If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
- [x] I've read the [Contributing guide](https://github.com/conan-io/conan/blob/develop2/.github/CONTRIBUTING.md).
- [x] I've followed the PEP8 style guides for Python code.
- [ ] I've opened another PR in the Conan docs repo to the ``develop`` branch, documenting this one.


uilianries: I did not check the code, but if I understand correctly, the uncompress status is updated every second. I just tried this branch and indeed it works until 99%, but Qt is huge, then, each step is repeated, but it's fine IMO:

```
qt/6.7.3: Calling source() in /home/uilian/.conan2/p/qte6ce713a78304/s/src
qt/6.7.3: Source ['https://download.qt.io/official_releases/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://download.qt.io/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirrors.ukfast.co.uk/sites/qt.io/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirrors.20i.com/pub/qt.io/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://ftp.nluug.nl/languages/qt/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirror.netcologne.de/qtproject/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://qt-mirror.dannhauer.de/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://ftp.fau.de/qtproject/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirrors.dotsrc.org/qtproject/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://ftp.icm.edu.pl/packages/qt/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://ftp.acc.umu.se/mirror/qt.io/qtproject/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://www.nic.funet.fi/pub/mirrors/download.qt-project.org/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://qt.mirror.constant.com/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirrors.sau.edu.cn/qt/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirrors.cloud.tencent.com/qt/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirror.bjtu.edu.cn/qt/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://mirrors.sjtug.sjtu.edu.cn/qt/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://ftp.jaist.ac.jp/pub/qtproject/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz', 'https://ftp.yz.yamagata-u.ac.jp/pub/qtproject/archive/qt/6.7/6.7.3/single/qt-everywhere-src-6.7.3.tar.xz'] retrieved from local download cache
qt/6.7.3: Unzipping qt-everywhere-src-6.7.3.tar.xz to /home/uilian/.conan2/p/qte6ce713a78304/s/src
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 6%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 12%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 16%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 17%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 18%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 19%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 20%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 20%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 21%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 22%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 23%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 25%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 25%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 25%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 25%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 25%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 26%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 26%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 26%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 26%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 27%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 28%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 29%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 30%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 31%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 33%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 35%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 37%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 38%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 39%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 40%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 41%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 41%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 41%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 42%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 43%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 44%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 44%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 45%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 45%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 45%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 45%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 45%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 46%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 47%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 47%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 48%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 50%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 50%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 51%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 51%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 52%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 52%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 53%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 53%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 54%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 55%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 55%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 56%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 57%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 58%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 59%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 60%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 60%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 61%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 63%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 65%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 66%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 67%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 69%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 70%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 72%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 72%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 72%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 72%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 73%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 73%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 73%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 74%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 75%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 75%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 76%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 78%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 78%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 79%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 79%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 80%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 81%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 82%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 82%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 83%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 84%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 85%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 85%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 86%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 86%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 86%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 87%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 87%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 87%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 87%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 89%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 90%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 91%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 92%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 92%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 93%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 94%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 95%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 96%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 96%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 96%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 97%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 97%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 98%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 98%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 98%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 99%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 99%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 99%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 99%
Uncompressing qt-everywhere-src-6.7.3.tar.xz: 99%
qt/6.7.3: Apply patch (bugfix): Workaround for too long .rps file name

-------- Installing package qt/6.7.3 (36 of 36) --------
qt/6.7.3: Building from source
```


perseoGI: @uilianries 
> indeed it works until 99%

Yes... I think I can add a final log which explicitly marks the 100% just for the TOC
> but Qt is huge, then, each step is repeated, but it's fine IMO:

That is a very good point, the progress printer is managed by a `TimedOutput` which has a print window based on the interval. We could opt for printing the progress only when the percentage has change since the last print but it may require some changes. Also, as we do not have a progress bar, printing the percentage regularly may be a good feedback for users to keep telling them that conan is doing some job and it has not gotten stuck. 


Patch:
diff --git a/codecov.yml b/codecov.yml
index 561bf760e..f418c3d13 100644
--- a/codecov.yml
+++ b/codecov.yml
@@ -1,1 +1,3 @@
 comment: false # Disable codecov PR comments -> leave only the checks
+ignore:
+  - "test/performance"
diff --git a/conan/tools/files/files.py b/conan/tools/files/files.py
index 433c30e2d..66a2a43a9 100644
--- a/conan/tools/files/files.py
+++ b/conan/tools/files/files.py
@@ -360,33 +360,37 @@ def untargz(filename, destination=".", pattern=None, strip_root=False, extract_f
         f = getattr(tarfile, f"{extract_filter}_filter", None) if extract_filter else None
         tarredgzippedFile.extraction_filter = f or (lambda member_, _: member_)
         if not pattern and not strip_root:
+            # Simple case: extract everything
             tarredgzippedFile.extractall(destination)
         else:
-            members = tarredgzippedFile.getmembers()
-
-            if pattern:
-                members = list(filter(lambda m: fnmatch(m.name, pattern),
-                                      tarredgzippedFile.getmembers()))
-            if strip_root:
-                names = [member.name.replace("\\", "/") for member in members]
-                common_folder = os.path.commonprefix(names).split("/", 1)[0]
-                if not common_folder and len(names) > 1:
-                    raise ConanException("The tgz file contains more than 1 folder in the root")
-                if len(names) == 1 and len(names[0].split("/", 1)) == 1:
-                    raise ConanException("The tgz file contains a file in the root")
-                # Remove the directory entry if present
-                members = [m for m in members if m.name != common_folder]
-                for member in members:
+            # Read members one by one and process them
+            common_folder = None
+            for member in tarredgzippedFile:
+                if pattern and not fnmatch(member.name, pattern):
+                    continue  # Skip files that don’t match the pattern
+
+                if strip_root:
                     name = member.name.replace("\\", "/")
-                    member.name = name.split("/", 1)[1]
+                    if not common_folder:
+                        splits = name.split("/", 1)
+                        # First case for a plain folder in the root
+                        if member.isdir() or len(splits) > 1:
+                            common_folder = splits[0]  # Find the root folder
+                        else:
+                            raise ConanException("Can't untar a tgz containing files in the root with strip_root enabled")
+                    if not name.startswith(common_folder):
+                        raise ConanException("The tgz file contains more than 1 folder in the root")
+
+                    # Adjust the member's name for extraction
+                    member.name = name[len(common_folder) + 1:]
                     member.path = member.name
-                    if member.linkpath.startswith(common_folder):
+                    if member.linkpath and member.linkpath.startswith(common_folder):
                         # https://github.com/conan-io/conan/issues/11065
-                        linkpath = member.linkpath.replace("\\", "/")
-                        member.linkpath = linkpath.split("/", 1)[1]
+                        member.linkpath = member.linkpath[len(common_folder) + 1:].replace("\\", "/")
                         member.linkname = member.linkpath
-            tarredgzippedFile.extractall(destination, members=members)
-
+                # Extract file one by one, avoiding `extractall()` with members parameter:
+                # This will avoid a first whole file read resulting in a performant improvement around 25%
+                tarredgzippedFile.extract(member, path=destination)
 
 def check_sha1(conanfile, file_path, signature):
     """
diff --git a/conan/tools/sbom/__init__.py b/conan/tools/sbom/__init__.py
index e69de29bb..1f1641e12 100644
--- a/conan/tools/sbom/__init__.py
+++ b/conan/tools/sbom/__init__.py
@@ -0,0 +1,1 @@
+from conan.tools.sbom.cyclonedx import cyclonedx_1_4
diff --git a/conan/tools/sbom/cyclonedx.py b/conan/tools/sbom/cyclonedx.py
index a875833df..967e45215 100644
--- a/conan/tools/sbom/cyclonedx.py
+++ b/conan/tools/sbom/cyclonedx.py
@@ -1,5 +1,6 @@
 
-def cyclonedx_1_4(graph, **kwargs):
+
+def cyclonedx_1_4(graph, add_build=False, add_tests=False, **kwargs):
     """
     (Experimental) Generate cyclone 1.4 sbom with json format
     """
@@ -9,8 +10,7 @@ def cyclonedx_1_4(graph, **kwargs):
 
     has_special_root_node = not (getattr(graph.root.ref, "name", False) and getattr(graph.root.ref, "version", False) and getattr(graph.root.ref, "revision", False))
     special_id = str(uuid.uuid4())
-
-    components = [node for node in graph.nodes]
+    components = [node for node in graph.nodes if (node.context == "host" or add_build) and (not node.test or add_tests)]
     if has_special_root_node:
         components = components[1:]
 
@@ -22,7 +22,9 @@ def cyclonedx_1_4(graph, **kwargs):
         dependencies.append(deps)
     for c in components:
         deps = {"ref": f"pkg:conan/{c.name}@{c.ref.version}?rref={c.ref.revision}"}
-        depends_on = [f"pkg:conan/{d.dst.name}@{d.dst.ref.version}?rref={d.dst.ref.revision}" for d in c.dependencies]
+        dep = [d for d in c.dependencies if (d.dst.context == "host" or add_build) and (not d.dst.test or add_tests)]
+
+        depends_on = [f"pkg:conan/{d.dst.name}@{d.dst.ref.version}?rref={d.dst.ref.revision}" for d in dep]
         if depends_on:
             deps["dependsOn"] = depends_on
         dependencies.append(deps)
diff --git a/conan/tools/scm/git.py b/conan/tools/scm/git.py
index 6e0ad9e27..c7bcb6ffe 100644
--- a/conan/tools/scm/git.py
+++ b/conan/tools/scm/git.py
@@ -260,15 +260,18 @@ class Git:
         files = files.splitlines()
         return files
 
-    def coordinates_to_conandata(self):
+    def coordinates_to_conandata(self, repository=False):
         """
         Capture the "url" and "commit" from the Git repo, calling ``get_url_and_commit()``, and then
         store those in the ``conandata.yml`` under the "scm" key. This information can be
         used later to clone and checkout the exact source point that was used to create this
         package, and can be useful even if the recipe uses ``exports_sources`` as mechanism to
         embed the sources.
+
+        :param repository: By default gets the commit of the defined folder, use repository=True to get
+                     the commit of the repository instead.
         """
-        scm_url, scm_commit = self.get_url_and_commit()
+        scm_url, scm_commit = self.get_url_and_commit(repository=repository)
         update_conandata(self._conanfile, {"scm": {"commit": scm_commit, "url": scm_url}})
 
     def checkout_from_conandata_coordinates(self):
diff --git a/conans/client/rest/file_uploader.py b/conans/client/rest/file_uploader.py
index 0b831d2da..5005a5c14 100644
--- a/conans/client/rest/file_uploader.py
+++ b/conans/client/rest/file_uploader.py
@@ -97,7 +97,7 @@ class FileUploader(object):
 
 
 class FileProgress(io.FileIO):
-    def __init__(self, path: str, msg: str = "Uploading", interval: float = 1, *args, **kwargs):
+    def __init__(self, path: str, msg: str = "Uploading", interval: float = 10, *args, **kwargs):
         super().__init__(path, *args, **kwargs)
         self._size = os.path.getsize(path)
         self._filename = os.path.basename(path)
diff --git a/conans/client/rest/rest_client_v2.py b/conans/client/rest/rest_client_v2.py
index 97ff28015..45ca3f192 100644
--- a/conans/client/rest/rest_client_v2.py
+++ b/conans/client/rest/rest_client_v2.py
@@ -277,27 +277,6 @@ class RestV2Methods:
             result.update({fn: os.path.join(dest_folder, fn) for fn in files})
         return result
 
-    @staticmethod
-    def _is_dir(path, files):
-        if path == ".":
-            return True
-        for the_file in files["files"]:
-            if path == the_file:
-                return False
-            elif the_file.startswith(path):
-                return True
-        raise NotFoundException("The specified path doesn't exist")
-
-    @staticmethod
-    def _list_dir_contents(path, files):
-        ret = []
-        for the_file in files["files"]:
-            if path == "." or the_file.startswith(path):
-                tmp = the_file[len(path) - 1:].split("/", 1)[0]
-                if tmp not in ret:
-                    ret.append(tmp)
-        return sorted(ret)
-
     def _upload_files(self, files, urls, ref):
         failed = []
         uploader = FileUploader(self.requester, self.verify_ssl, self._config)
diff --git a/conans/client/rest/rest_routes.py b/conans/client/rest/rest_routes.py
index 7dab1f077..b2059162d 100644
--- a/conans/client/rest/rest_routes.py
+++ b/conans/client/rest/rest_routes.py
@@ -31,22 +31,10 @@ class RestRoutes(object):
     def recipe_revision_file(self):
         return '%s/files/{path}' % self.recipe_revision
 
-    @property
-    def packages(self):
-        return '%s/packages' % self.recipe
-
     @property
     def packages_revision(self):
         return '%s/packages' % self.recipe_revision
 
-    @property
-    def package(self):
-        return '%s/{package_id}' % self.packages
-
-    @property
-    def package_files(self):
-        return '%s/files' % self.package
-
     @property
     def package_recipe_revision(self):
         """Route for a package specifying the recipe revision but not the package revision"""


Test patch:
diff --git a/conan/test/assets/__init__.py b/conan/test/assets/__init__.py
index c7f40f86e..e69de29bb 100644
--- a/conan/test/assets/__init__.py
+++ b/conan/test/assets/__init__.py
@@ -1,16 +0,0 @@
-import os
-import shutil
-
-
-def copy_assets(src_folder, dst_folder, assets=None):
-    assets_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "files")
-    if src_folder:
-        src_folder = os.path.join(assets_path, src_folder)
-    assets = assets or os.listdir(src_folder)
-    for asset in assets:
-        s = os.path.join(src_folder, asset)
-        d = os.path.join(dst_folder, asset)
-        if os.path.isdir(s):
-            shutil.copytree(s, d)
-        else:
-            shutil.copy2(s, d)
diff --git a/conan/test/assets/visual_project_files.py b/conan/test/assets/visual_project_files.py
index 5db7b7cec..9b9f5ceb5 100644
--- a/conan/test/assets/visual_project_files.py
+++ b/conan/test/assets/visual_project_files.py
@@ -245,47 +245,9 @@ int main()
 }
 '''
 
-main_file_17_2015 = '''
-//available in 2015 with c++17
-#include <iostream>
-
-namespace test::mytest {
-struct mystruct {};
-};
-int main(){
-test::mytest::mystruct t;
-}
-'''
-
-
-main_file_17_2017 = '''
-//only available in visual 2017 with c++17
-#include <iostream>
-
-auto hello(){
-    return 2;
-}
-
-int main()
-{
-    if constexpr(true)
-        std::cout << "Hello World!" << std::endl;
-    return 0;
-}
-'''
-
-
-def get_vs_project_files(std=False):
-    if not std:
-        main = main_file
-    elif std == "cpp17_2015":
-        main = main_file_17_2015
-    elif std == "cpp17_2017":
-        main = main_file_17_2017
-    else:
-        raise Exception("Incorrect std")
 
+def get_vs_project_files():
     return {"MyProject.sln": sln_file,
             "MyProject/MyProject.vcxproj": vcxproj_file,
             "MyProject/MyProject.vcxproj.filters": filters_file,
-            "MyProject/main.cpp": main}
+            "MyProject/main.cpp": main_file}
diff --git a/conan/test/utils/mocks.py b/conan/test/utils/mocks.py
index 9638b4ad8..cd4e9f45e 100644
--- a/conan/test/utils/mocks.py
+++ b/conan/test/utils/mocks.py
@@ -9,25 +9,6 @@ from conan.internal.model.layout import Folders, Infos
 from conan.internal.model.options import Options
 
 
-class LocalDBMock(object):
-
-    def __init__(self, user=None, access_token=None, refresh_token=None):
-        self.user = user
-        self.access_token = access_token
-        self.refresh_token = refresh_token
-
-    def get_login(self, _):
-        return self.user, self.access_token, self.refresh_token
-
-    def get_username(self, _):
-        return self.user
-
-    def store(self, user, access_token, refresh_token, _):
-        self.user = user
-        self.access_token = access_token
-        self.refresh_token = refresh_token
-
-
 class RedirectedInputStream:
     """
     Mock for testing. If get_username or get_password is requested will raise
@@ -67,19 +48,6 @@ class MockSettings(object):
         return defaultdict(lambda: [])
 
 
-class MockCppInfo(object):
-    def __init__(self):
-        self.bin_paths = []
-        self.lib_paths = []
-        self.include_paths = []
-        self.libs = []
-        self.cflags = []
-        self.cxxflags = []
-        self.defines = []
-        self.frameworks = []
-        self.framework_paths = []
-
-
 class ConanFileMock(ConanFile):
     def __init__(self, settings=None, options=None, runner=None, display_name=""):
         self.display_name = display_name
diff --git a/conan/test/utils/tools.py b/conan/test/utils/tools.py
index b143a7502..d0c44a9ea 100644
--- a/conan/test/utils/tools.py
+++ b/conan/test/utils/tools.py
@@ -41,7 +41,6 @@ from conan.api.model import PkgReference
 from conan.internal.model.profile import Profile
 from conan.api.model import RecipeReference
 from conan.internal.model.settings import Settings
-from conan.test.assets import copy_assets
 from conan.test.assets.genconanfile import GenConanfile
 from conan.test.utils.artifactory import ArtifactoryServer
 from conan.test.utils.mocks import RedirectedInputStream
@@ -661,9 +660,6 @@ class TestClient:
     def save_home(self, files):
         self.save(files, path=self.cache_folder)
 
-    def copy_assets(self, origin_folder, assets=None):
-        copy_assets(origin_folder, self.current_folder, assets)
-
     # Higher level operations
     def remove_all(self):
         self.run("remove '*' -c")
diff --git a/test/functional/sbom/test_cyclonedx.py b/test/functional/sbom/test_cyclonedx.py
index 8aafd1945..3c1b80a0b 100644
--- a/test/functional/sbom/test_cyclonedx.py
+++ b/test/functional/sbom/test_cyclonedx.py
@@ -13,10 +13,10 @@ import json
 import os
 from conan.errors import ConanException
 from conan.api.output import ConanOutput
-from conan.tools.sbom.cyclonedx import cyclonedx_1_4
+from conan.tools.sbom import cyclonedx_1_4
 
 def post_package(conanfile):
-    sbom_cyclonedx_1_4 = cyclonedx_1_4(conanfile.subgraph)
+    sbom_cyclonedx_1_4 = cyclonedx_1_4(conanfile.subgraph, add_build=%s, add_tests=%s)
     metadata_folder = conanfile.package_metadata_folder
     file_name = "sbom.cdx.json"
     with open(os.path.join(metadata_folder, file_name), 'w') as f:
@@ -28,14 +28,28 @@ def post_package(conanfile):
 def hook_setup_post_package():
     tc = TestClient()
     hook_path = os.path.join(tc.paths.hooks_path, "hook_sbom.py")
-    save(hook_path, sbom_hook_post_package)
+    save(hook_path, sbom_hook_post_package % ("True", "True"))
+    return tc
+
+@pytest.fixture()
+def hook_setup_post_package_no_tool_requires():
+    tc = TestClient()
+    hook_path = os.path.join(tc.paths.hooks_path, "hook_sbom.py")
+    save(hook_path, sbom_hook_post_package % ("False", "True"))
+    return tc
+
+@pytest.fixture()
+def hook_setup_post_package_no_test():
+    tc = TestClient()
+    hook_path = os.path.join(tc.paths.hooks_path, "hook_sbom.py")
+    save(hook_path, sbom_hook_post_package % ("True", "False"))
     return tc
 
 @pytest.fixture()
 def hook_setup_post_package_tl(transitive_libraries):
     tc = transitive_libraries
     hook_path = os.path.join(tc.paths.hooks_path, "hook_sbom.py")
-    save(hook_path, sbom_hook_post_package)
+    save(hook_path, sbom_hook_post_package % ("True", "True"))
     return tc
 
 
@@ -65,6 +79,55 @@ def test_sbom_generation_skipped_dependencies(hook_setup_post_package):
     # A skipped dependency also shows up in the sbom
     assert "pkg:conan/dep@1.0?rref=6a99f55e933fb6feeb96df134c33af44" in content
 
+def test_sbom_generation_no_tool_requires(hook_setup_post_package_no_tool_requires):
+    tc = hook_setup_post_package_no_tool_requires
+    tc.save({"app/conanfile.py": GenConanfile("app", "1.0")
+                                .with_package_type("application"),
+             "conanfile.py": GenConanfile("foo", "1.0").with_tool_requires("app/1.0")})
+    tc.run("create app")
+    tc.run("create .")
+    create_layout = tc.created_layout()
+
+    cyclone_path = os.path.join(create_layout.metadata(), "sbom.cdx.json")
+    content = tc.load(cyclone_path)
+
+    assert "pkg:conan/app" not in content
+
+def test_sbom_generation_transitive_test_requires(hook_setup_post_package_no_test):
+    tc = hook_setup_post_package_no_test
+    tc.save({"test_re/conanfile.py": GenConanfile("test_re", "1.0"),
+             "app/conanfile.py": GenConanfile("app", "1.0")
+                                .with_package_type("application")
+                                .with_test_requires("test_re/1.0"),
+             "conanfile.py": GenConanfile("foo", "1.0").with_tool_requires("app/1.0")})
+    tc.run("create test_re")
+
+    tc.run("create app")
+    create_layout = tc.created_layout()
+    cyclone_path = os.path.join(create_layout.metadata(), "sbom.cdx.json")
+    content = tc.load(cyclone_path)
+    assert "pkg:conan/test_re@1.0" not in content
+
+    tc.run("create .")
+    create_layout = tc.created_layout()
+    cyclone_path = os.path.join(create_layout.metadata(), "sbom.cdx.json")
+    content = tc.load(cyclone_path)
+    assert "pkg:conan/test_re@1.0" not in content
+
+def test_sbom_generation_dependency_test_require(hook_setup_post_package_no_test):
+    tc = hook_setup_post_package_no_test
+    tc.save({"special/conanfile.py": GenConanfile("special", "1.0"),
+             "foo/conanfile.py": GenConanfile("foo", "1.0")
+            .with_test_requires("special/1.0"),
+             "conanfile.py": GenConanfile("bar", "1.0").with_tool_requires("foo/1.0").with_require("special/1.0")})
+    tc.run("create special")
+    tc.run("create foo")
+
+    tc.run("create .")
+    create_layout = tc.created_layout()
+    cyclone_path = os.path.join(create_layout.metadata(), "sbom.cdx.json")
+    content = tc.load(cyclone_path)
+    assert "pkg:conan/special@1.0" in content
 
 # Using the sbom tool with "conan install"
 sbom_hook_post_generate = """
@@ -72,7 +135,7 @@ import json
 import os
 from conan.errors import ConanException
 from conan.api.output import ConanOutput
-from conan.tools.sbom.cyclonedx import cyclonedx_1_4
+from conan.tools.sbom import cyclonedx_1_4
 
 def post_generate(conanfile):
     sbom_cyclonedx_1_4 = cyclonedx_1_4(conanfile.subgraph)
diff --git a/test/functional/toolchains/microsoft/test_msbuild.py b/test/functional/toolchains/microsoft/test_msbuild.py
index c476e07c0..0754e2e33 100644
--- a/test/functional/toolchains/microsoft/test_msbuild.py
+++ b/test/functional/toolchains/microsoft/test_msbuild.py
@@ -488,12 +488,12 @@ class TestWin:
         check_vs_runtime("Release/MyApp.exe", client, ide_version, build_type="Release",
                          static_runtime=static_runtime)
 
-    @pytest.mark.tool("cmake")
-    @pytest.mark.tool("visual_studio", "16")
+    @pytest.mark.tool("cmake", "3.23")
+    @pytest.mark.tool("visual_studio", "17")
     def test_toolchain_win_debug(self):
         client = TestClient(path_with_spaces=False)
         settings = [("compiler",  "msvc"),
-                    ("compiler.version",  "192"),
+                    ("compiler.version",  "193"),
                     ("compiler.runtime",  "dynamic"),
                     ("build_type",  "Debug"),
                     ("arch",  "x86_64")]
@@ -514,27 +514,27 @@ class TestWin:
         # Run the configure corresponding to this test case
         client.run("build . %s" % (settings, ))
         assert "conanfile.py: MSBuildToolchain created conantoolchain_debug_x64.props" in client.out
-        assert f"conanvcvars.bat: Activating environment Visual Studio 16" in client.out
+        assert f"conanvcvars.bat: Activating environment Visual Studio 17" in client.out
         assert "[vcvarsall.bat] Environment initialized for: 'x64'" in client.out
         self._run_app(client, "x64", "Debug")
         assert "Hello World Debug" in client.out
-        check_exe_run(client.out, "main", "msvc", "192", "Debug", "x86_64", "14",
+        check_exe_run(client.out, "main", "msvc", "19", "Debug", "x86_64", "14",
                       {"DEFINITIONS_BOTH": 'True',
                        "DEFINITIONS_BOTH2": "True",
                        "DEFINITIONS_BOTH_INT": "123",
                        "DEFINITIONS_CONFIG": 'Debug',
                        "DEFINITIONS_CONFIG2": 'Debug',
                        "DEFINITIONS_CONFIG_INT": "234"})
-        check_vs_runtime("x64/Debug/MyApp.exe", client, "16", build_type="Debug")
+        check_vs_runtime("x64/Debug/MyApp.exe", client, "17", build_type="Debug")
 
-    @pytest.mark.tool("cmake")
-    @pytest.mark.tool("visual_studio", "16")
+    @pytest.mark.tool("cmake", "3.23")
+    @pytest.mark.tool("visual_studio", "17")
     def test_toolchain_win_multi(self):
-        ide_version = "16"
+        ide_version = "17"
         client = TestClient(path_with_spaces=False)
 
         settings = [("compiler", "msvc"),
-                    ("compiler.version", "192"),
+                    ("compiler.version", "193"),
                     ("compiler.cppstd", "17"),
                     ("compiler.runtime", "static")]
 
@@ -582,7 +582,7 @@ class TestWin:
             assert "[vcvarsall.bat] Environment initialized for: 'x64'" in client.out
 
             self._run_app(client, arch, build_type, shared)
-            check_exe_run(client.out, "main", "msvc", "192", build_type, arch, "17",
+            check_exe_run(client.out, "main", "msvc", "19", build_type, arch, "17",
                           {"DEFINITIONS_BOTH": "True",
                            "DEFINITIONS_CONFIG": build_type})
 
diff --git a/test/integration/conanfile/runner_test.py b/test/integration/conanfile/runner_test.py
index add796687..28ce9c9b9 100644
--- a/test/integration/conanfile/runner_test.py
+++ b/test/integration/conanfile/runner_test.py
@@ -1,20 +1,9 @@
-import os
 import textwrap
-import unittest
 
 from conan.test.utils.tools import TestClient
 
 
-class RunnerTest(unittest.TestCase):
-
-    def _install_and_build(self, client, conanfile_text):
-        files = {"conanfile.py": conanfile_text}
-        test_folder = os.path.join(client.current_folder, "test_folder")
-        self.assertFalse(os.path.exists(test_folder))
-        client.save(files)
-        client.run("install .")
-        client.run("build .")
-        return client
+class TestRunner:
 
     def test_ignore_error(self):
         conanfile = """from conan import ConanFile
@@ -26,7 +15,7 @@ class Pkg(ConanFile):
         client = TestClient()
         client.save({"conanfile.py": conanfile})
         client.run("source .")
-        self.assertIn("RETCODE True", client.out)
+        assert "RETCODE True" in client.out
 
     def test_runner_capture_output(self):
         conanfile = textwrap.dedent("""
@@ -38,7 +27,7 @@ class Pkg(ConanFile):
         client = TestClient()
         client.save({"conanfile.py": conanfile})
         client.run("source .")
-        self.assertIn("hello Conan!", client.out)
+        assert "hello Conan!" in client.out
 
     def test_custom_stream_error(self):
         # https://github.com/conan-io/conan/issues/7888
@@ -54,7 +43,7 @@ class Pkg(ConanFile):
         client = TestClient()
         client.save({"conanfile.py": conanfile})
         client.run("source .")
-        self.assertIn('conanfile.py: Buffer got msgs Hello', client.out)
+        assert 'conanfile.py: Buffer got msgs Hello' in client.out
 
     def test_custom_stream_stderr(self):
         conanfile = textwrap.dedent("""
@@ -69,4 +58,4 @@ class Pkg(ConanFile):
         client = TestClient()
         client.save({"conanfile.py": conanfile})
         client.run("source .")
-        self.assertIn('conanfile.py: Buffer got stderr msgs Hello', client.out)
+        assert 'conanfile.py: Buffer got stderr msgs Hello' in client.out
diff --git a/test/integration/environment/test_env.py b/test/integration/environment/test_env.py
index a3b506ee6..293371d28 100644
--- a/test/integration/environment/test_env.py
+++ b/test/integration/environment/test_env.py
@@ -110,14 +110,14 @@ def test_complete(client, gtest_run_true):
     client.run("install . -s:b os=Windows -s:h os=Linux --build=missing")
     # Run the BUILD environment
     if platform.system() == "Windows":
-        cmd = environment_wrap_command(ConanFileMock(),"conanbuildenv", client.current_folder, "mycmake.bat")
+        cmd = environment_wrap_command(ConanFileMock(), "conanbuildenv", client.current_folder, "mycmake.bat")
         client.run_command(cmd)
         assert "MYCMAKE=Windows!!" in client.out
         assert "MYOPENSSL=Windows!!" in client.out
 
     # Run the RUN environment
     if platform.system() != "Windows":
-        cmd = environment_wrap_command(ConanFileMock(),"conanrunenv", client.current_folder,
+        cmd = environment_wrap_command(ConanFileMock(), "conanrunenv", client.current_folder,
                                        "mygtest.sh && .{}myrunner.sh".format(os.sep))
         client.run_command(cmd, assert_error=not gtest_run_true)
         if gtest_run_true:
@@ -210,7 +210,7 @@ def test_profile_buildenv():
     client.run("install . -pr=myprofile")
     # Run the BUILD environment
     ext = "bat" if platform.system() == "Windows" else "sh"  # TODO: Decide on logic .bat vs .sh
-    cmd = environment_wrap_command(ConanFileMock(),"conanbuildenv", client.current_folder,
+    cmd = environment_wrap_command(ConanFileMock(), "conanbuildenv", client.current_folder,
                                    "mycompiler.{}".format(ext))
     client.run_command(cmd)
     assert "MYCOMPILER!!" in client.out
@@ -621,14 +621,13 @@ def test_massive_paths(num_deps):
     assert os.path.isfile(os.path.join(client.current_folder, "conanrunenv.ps1"))
     assert not os.path.isfile(os.path.join(client.current_folder, "conanrunenv.bat"))
     for i in range(num_deps):
-        cmd = environment_wrap_command(ConanFileMock(),"conanrunenv", client.current_folder,
+        cmd = environment_wrap_command(ConanFileMock(), "conanrunenv", client.current_folder,
                                        "mycompiler{}.bat".format(i))
-        if num_deps > 50:  # to be safe if we change the "num_deps" number
-            client.run_command(cmd, assert_error=True)
-            assert "is not recognized as an internal" in client.out
-        else:
-            client.run_command(cmd)
-            assert "MYTOOL {}!!".format(i) in client.out
+        # if num_deps > 50:  # to be safe if we change the "num_deps" number
+        #    client.run_command(cmd, assert_error=True)
+        #    assert "is not recognized as an internal" in client.out
+        client.run_command(cmd)
+        assert "MYTOOL {}!!".format(i) in client.out
 
     # Test .bats now
     client.save({"conanfile.py": conanfile}, clean_first=True)
@@ -636,14 +635,14 @@ def test_massive_paths(num_deps):
     assert not os.path.isfile(os.path.join(client.current_folder, "conanrunenv.ps1"))
     assert os.path.isfile(os.path.join(client.current_folder, "conanrunenv.bat"))
     for i in range(num_deps):
-        cmd = environment_wrap_command(ConanFileMock(),"conanrunenv", client.current_folder,
+        cmd = environment_wrap_command(ConanFileMock(), "conanrunenv", client.current_folder,
                                        "mycompiler{}.bat".format(i))
-        if num_deps > 50:  # to be safe if we change the "num_deps" number
-            client.run_command(cmd, assert_error=True)
-            # This also fails, but without an error message (in my terminal, it kills the terminal!)
-        else:
-            client.run_command(cmd)
-            assert "MYTOOL {}!!".format(i) in client.out
+        # if num_deps > 50:  # to be safe if we change the "num_deps" number
+        #    client.run_command(cmd, assert_error=True)
+        #    # This also fails, but without an error message (in my terminal, it kills the terminal!)
+        # else:
+        client.run_command(cmd)
+        assert "MYTOOL {}!!".format(i) in client.out
 
 
 def test_profile_build_env_spaces():
diff --git a/test/integration/graph/core/graph_manager_base.py b/test/integration/graph/core/graph_manager_base.py
index 97d85b96f..23dcdd9df 100644
--- a/test/integration/graph/core/graph_manager_base.py
+++ b/test/integration/graph/core/graph_manager_base.py
@@ -67,7 +67,7 @@ class GraphManagerTest(unittest.TestCase):
         self._cache_recipe(ref, conanfile)
 
     @staticmethod
-    def recipe_consumer(reference=None, requires=None, build_requires=None, tool_requires=None):
+    def recipe_consumer(reference=None, requires=None, build_requires=None):
         path = temp_folder()
         path = os.path.join(path, "conanfile.py")
         conanfile = GenConanfile()
@@ -80,9 +80,6 @@ class GraphManagerTest(unittest.TestCase):
         if build_requires:
             for r in build_requires:
                 conanfile.with_build_requires(r)
-        if tool_requires:
-            for r in tool_requires:
-                conanfile.with_tool_requires(r)
         save(path, str(conanfile))
         return path
 
diff --git a/test/integration/graph/core/graph_manager_test.py b/test/integration/graph/core/graph_manager_test.py
index 095dfcc70..4e226b7bb 100644
--- a/test/integration/graph/core/graph_manager_test.py
+++ b/test/integration/graph/core/graph_manager_test.py
@@ -1,7 +1,6 @@
 import pytest
 from parameterized import parameterized
 
-from conans.client.graph.graph import BINARY_SKIP
 from conans.client.graph.graph_error import GraphMissingError, GraphLoopError, GraphConflictError
 from conan.errors import ConanException
 from test.integration.graph.core.graph_manager_base import GraphManagerTest
@@ -11,24 +10,17 @@ from conan.test.utils.tools import GenConanfile
 def _check_transitive(node, transitive_deps):
     values = list(node.transitive_deps.values())
 
-    if len(values) != len(transitive_deps):
-        values = [r.require.ref for r in values]
-        raise Exception("{}: Number of deps don't match \n{}!=\n{}".format(node, values,
-                                                                           transitive_deps))
+    assert len(values) == len(transitive_deps), f"{node}: Number of deps don't match " \
+                                                f"\n{[r.require.ref for r in values]}" \
+                                                f"!=\n{transitive_deps}"
 
     for v1, v2 in zip(values, transitive_deps):
-        if v1.node is not v2[0]:
-            raise Exception(f"{v1.node}!={v2[0]}")
-        if v1.require.headers is not v2[1]:
-            raise Exception(f"{v1.node}!={v2[0]} headers")
-        if v1.require.libs is not v2[2]:
-            raise Exception(f"{v1.node}!={v2[0]} libs")
-        if v1.require.build is not v2[3]:
-            raise Exception(f"{v1.node}!={v2[0]} build")
-        if v1.require.run is not v2[4]:
-            raise Exception(f"{v1.node}!={v2[0]} run")
-        if len(v2) > 5:
-            assert v1.require.package_id_mode is v2[5]
+        assert v1.node is v2[0], f"{v1.node}!={v2[0]}"
+        assert v1.require.headers is v2[1], f"{v1.node}!={v2[0]} headers"
+        assert v1.require.libs is v2[2], f"{v1.node}!={v2[0]} libs"
+        assert v1.require.build is v2[3], f"{v1.node}!={v2[0]} build"
+        assert v1.require.run is  v2[4], f"{v1.node}!={v2[0]} run"
+        assert len(v2) <= 5
 
 
 class TestLinear(GraphManagerTest):
diff --git a/test/integration/remote/auth_bearer_test.py b/test/integration/remote/auth_bearer_test.py
index b5d05e531..19008a2c9 100644
--- a/test/integration/remote/auth_bearer_test.py
+++ b/test/integration/remote/auth_bearer_test.py
@@ -1,20 +1,10 @@
-import unittest
-
 from bottle import request
 
+from conan.test.assets.genconanfile import GenConanfile
 from conan.test.utils.tools import TestClient, TestServer
 
 
-conanfile = """
-from conan import ConanFile
-
-class OpenSSLConan(ConanFile):
-    name = "hello"
-    version = "0.1"
-"""
-
-
-class AuthorizationHeaderSpy(object):
+class AuthorizationHeaderSpy:
     """ Generic plugin to handle Authorization header. Must be extended and implement
     some abstract methods in subclasses"""
 
@@ -24,50 +14,32 @@ class AuthorizationHeaderSpy(object):
     def __init__(self):
         self.auths = []
 
-    def apply(self, callback, context):  # @UnusedVariable
+    def apply(self, callback, context):  # noqa
         auth = request.headers.get("Authorization")
         name = callback.__name__
         self.auths.append((name, auth))
         return callback
 
 
-class ReturnHandlerPlugin(object):
-
-    name = 'ReturnHandlerPluginSpy'
-    api = 2
-
-    def apply(self, callback, _):
-        """Apply plugin"""
-        def wrapper(*args, **kwargs):
-            """Capture possible exceptions to manage the return"""
-            result = callback(*args, **kwargs)
-            if isinstance(result, dict):
-                for k, v in result.items():
-                    result[k] = v.split("?signature=")[0]
-            return result
-        return wrapper
-
-
-class AuthorizeBearerTest(unittest.TestCase):
+class TestAuthorizeBearer:
 
     def test_basic(self):
         auth = AuthorizationHeaderSpy()
         server = TestServer(plugins=[auth])
-        servers = {"default": server}
-        client = TestClient(servers=servers, inputs=["admin", "password"])
-        client.save({"conanfile.py": conanfile})
+        client = TestClient(servers={"default": server}, inputs=["admin", "password"])
+        client.save({"conanfile.py": GenConanfile("hello", "0.1")})
         client.run("export . --user=lasote --channel=stable")
         errors = client.run("upload hello/0.1@lasote/stable -r default --only-recipe")
-        self.assertFalse(errors)
+        assert not errors
 
         expected_calls = [('get_recipe_revisions_references', None),
                           ('check_credentials', None),
                           ('authenticate', 'Basic'),
                           ('upload_recipe_file', 'Bearer')]
 
-        self.assertEqual(len(expected_calls), len(auth.auths))
+        assert len(expected_calls) == len(auth.auths)
         for i, (method, auth_type) in enumerate(expected_calls):
             real_call = auth.auths[i]
-            self.assertEqual(method, real_call[0])
+            assert method == real_call[0]
             if auth_type:
-                self.assertIn(auth_type, real_call[1])
+                assert auth_type in real_call[1]
diff --git a/test/unittests/tools/files/test_downloads.py b/test/unittests/tools/files/test_downloads.py
index 165001b46..25019d08a 100644
--- a/test/unittests/tools/files/test_downloads.py
+++ b/test/unittests/tools/files/test_downloads.py
@@ -1,5 +1,6 @@
 import os
 import shutil
+from unittest import mock
 
 import pytest
 
@@ -13,27 +14,31 @@ from conan.test.utils.tools import redirect_output, TestRequester
 from conans.util.files import save, load, chdir, mkdir
 
 
-@pytest.mark.skip(reason="This causes more troubles than benefits, external ftp download is testing "
-                         "very little conan code, mostly python")
+@mock.patch('ftplib.FTP', autospec=True)
 class TestFTP:
 
-    def test_ftp_auth(self):
-        filename = "/pub/example/readme.txt"
-        conanfile = ConanFileMock()
-        ftp_download(conanfile, "test.rebex.net", filename, "demo", "password")
-        assert os.path.exists(os.path.basename(filename))
-
-    def test_ftp_invalid_path(self):
+    def test_ftp_auth(self, ftp_mock):
+        ftp_object = ftp_mock.return_value
+        filename = "/pub/example/readme2.txt"
+        with chdir(temp_folder()):  # To not pollute the project
+            ftp_download(None, "test.rebex.net", filename, "demo", "password")
+        ftp_mock.assert_called_with('test.rebex.net')
+        assert ftp_object.login.called
+        ftp_object.cwd.assert_called_with('/pub/example')
+
+    def test_ftp_invalid_path(self, ftp_mock):
+        ftp_object = ftp_mock.return_value
+        ftp_object.cwd.side_effect = Exception("550 The system cannot find the file specified.")
         with pytest.raises(ConanException) as exc:
-            conanfile = ConanFileMock()
-            ftp_download(conanfile, "test.rebex.net", "invalid-file", "demo", "password")
+            ftp_download(None, "test.rebex.net", "/path/invalid-file", "demo", "password")
         assert "550 The system cannot find the file specified." in str(exc.value)
         assert not os.path.exists("invalid-file")
 
-    def test_ftp_invalid_auth(self):
+    def test_ftp_invalid_auth(self, ftp_mock):
+        ftp_object = ftp_mock.return_value
+        ftp_object.login.side_effect = Exception("530 User cannot log in.")
         with pytest.raises(ConanException) as exc:
-            conanfile = ConanFileMock()
-            ftp_download(conanfile, "test.rebex.net", "readme.txt", "demo", "invalid")
+            ftp_download(None, "test.rebex.net", "readme.txt", "demo", "invalid")
         assert "530 User cannot log in." in str(exc.value)
         assert not os.path.exists("readme.txt")
 
diff --git a/test/unittests/tools/files/test_zipping.py b/test/unittests/tools/files/test_zipping.py
index 0c41a494d..655271ad5 100644
--- a/test/unittests/tools/files/test_zipping.py
+++ b/test/unittests/tools/files/test_zipping.py
@@ -137,7 +137,7 @@ def test_untargz_with_strip_root_fails():
     dest_dir = temp_folder()
     with pytest.raises(ConanException) as error:
         unzip(conanfile, archive, dest_dir, strip_root=True)
-    assert "The tgz file contains more than 1 folder in the root" in str(error.value)
+    assert "Can't untar a tgz containing files in the root with strip_root enabled" in str(error.value)
 
 
 def test_untargz_with_strip_root_and_pattern():
diff --git a/test/unittests/tools/gnu/autotoolschain_test.py b/test/unittests/tools/gnu/autotoolschain_test.py
index eec8d7cce..43d79fc43 100644
--- a/test/unittests/tools/gnu/autotoolschain_test.py
+++ b/test/unittests/tools/gnu/autotoolschain_test.py
@@ -126,9 +126,8 @@ def test_get_gnu_triplet_for_cross_building_raise_error():
     conanfile.settings_build = MockSettings({"os": "Solaris", "arch": "x86"})
     with pytest.raises(ConanException) as conan_error:
         AutotoolsToolchain(conanfile)
-        msg = "'compiler' parameter for 'get_gnu_triplet()' is not specified and " \
-              "needed for os=Windows"
-        assert msg == str(conan_error.value)
+    msg = "'compiler' parameter for 'get_gnu_triplet()' is not specified and needed for os=Windows"
+    assert msg == str(conan_error.value)
 
 
 def test_compilers_mapping():
diff --git a/test/unittests/tools/gnu/test_gnutoolchain.py b/test/unittests/tools/gnu/test_gnutoolchain.py
index 48fffbe94..b18e2119f 100644
--- a/test/unittests/tools/gnu/test_gnutoolchain.py
+++ b/test/unittests/tools/gnu/test_gnutoolchain.py
@@ -124,9 +124,8 @@ def test_get_gnu_triplet_for_cross_building_raise_error():
     conanfile.settings_build = MockSettings({"os": "Solaris", "arch": "x86"})
     with pytest.raises(ConanException) as conan_error:
         GnuToolchain(conanfile)
-        msg = "'compiler' parameter for 'get_gnu_triplet()' is not specified and " \
-              "needed for os=Windows"
-        assert msg == str(conan_error.value)
+    msg = "'compiler' parameter for 'get_gnu_triplet()' is not specified and needed for os=Windows"
+    assert msg == str(conan_error.value)
 
 
 def test_compilers_mapping():
diff --git a/test/unittests/util/files/strip_root_extract_test.py b/test/unittests/util/files/strip_root_extract_test.py
index 3f8746b99..d65bcb367 100644
--- a/test/unittests/util/files/strip_root_extract_test.py
+++ b/test/unittests/util/files/strip_root_extract_test.py
@@ -255,5 +255,20 @@ class TarExtractPlainTest(unittest.TestCase):
 
         # Extract without the subfolder
         extract_folder = temp_folder()
-        with self.assertRaisesRegex(ConanException, "The tgz file contains a file in the root"):
+        with self.assertRaisesRegex(ConanException, "Can't untar a tgz containing files in the root with strip_root enabled"):
+            unzip(ConanFileMock(), tgz_file, destination=extract_folder, strip_root=True)
+
+    def test_invalid_flat_multiple_file(self):
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            save("file1", "contentsfile1")
+            save("file2", "contentsfile2")
+
+        zip_folder = temp_folder()
+        tgz_file = os.path.join(zip_folder, "file.tar.gz")
+        self._compress_folder(tmp_folder, tgz_file)
+
+        # Extract without the subfolder
+        extract_folder = temp_folder()
+        with self.assertRaisesRegex(ConanException, "Can't untar a tgz containing files in the root with strip_root enabled"):
             unzip(ConanFileMock(), tgz_file, destination=extract_folder, strip_root=True)