Repository: jazzband/pip-tools

Issue:
Request: Compact annotations
#### What's the problem this feature will solve?
**Primary:**

I would like to see a lot more information on my screen without having scroll up and down, when viewing annotated `pip-compile` output. At least as much as was the case before annotations became split into many lines.

**Secondary:**

I would like to use simple line-oriented shell tools to casually inspect properties of the `pip-compile` output, without resorting to multi-line regular expressions.

#### Describe the solution you'd like
I would like the ability, through a flag or any other means, to have `pip-compile`'s annotated output place annotations for a pinned requirement on the same line as said requirement.

My screen is wide and short, as is common for computers. It's a more efficient use of this space to use more width and less height.

Example of the different use of space, for the `magic-wormhole` package from PyPI:

![Example](https://user-images.githubusercontent.com/1787385/103960545-629a1300-5120-11eb-9a6b-7bbf41292d64.png)

#### Alternative Solutions
- I could skip annotations, but then I'd lose a lot of valuable information.
- I could add a post-processor to re-write the `pip-compile` output every time I compile.
- I could revert #1237 and poorly maintain a fork of this project, but that would make me sad.

#### Additional context
- #1297
- #1237
- #1377


Discussion:
Based on feedback it looks like we should have introduced one-per-line annotations via an option. Well, better to do it late than never. I don't mind adding an option for the old-style annotations. 

Just to give some feedback in the other direction: I'm very happy with the new separate line mode. The old format often caused really annoying merge conflicts that more than once lead developers in our team to just "blow away" the generated requirements.txt and compile a new one (with the obvious drawbacks of uncontrolled version changes)...
@jdufresne 

Would you like to tackle this one?
As this is an issue for enough users, please go ahead with adding an option. While I will personally not use it, clearly enough people will so might as well try to make this useful for as many people as possible.
It occurs to me that this, as well as a number of other issues in `pip-tools`'s tracker that want to alter and customize `pip-tools`'s output in various ways, could perhaps be enabled generally if `pip-compile` provided the option of outputting the information it has in a structured format like json. Then it would be a lot easier for people to create `requirements.txt` files in the format (and content) they want if not explicitly supported by `pip-compile`. Currently, trying to alter `pip-compile`'s output by parsing is fragile because the output format can change from release to release.
@cjerdonek 
If I got you right, then I disagree. I think `--format` with some argument is classic solution and enough for this case.
I think https://github.com/jazzband/pip-tools/pull/1297#issuecomment-758588222 is the way to go, future proof.
> It occurs to me that this, as well as a number of other issues in pip-tools's tracker that want to alter and customize pip-tools's output in various ways, could perhaps be enabled generally if pip-compile provided the option of outputting the information it has in a structured format like json. 

FYI, I just created issue #1377 expanding on my suggestion above.

> If I got you right, then I disagree. I think --format with some argument is classic solution and enough for this case.

@pohmelie I don't disagree with you. `pip-tools` can still implement feature requests like this. I was merely pointing out a way that `pip-tools` could make it easier for users to work-around issues in cases where `pip-tools` hasn't yet implemented the feature in `pip-tools` itself.

@cjerdonek, that sounds nice.
JSON output seems potentially useful but orthogonal to the issue at hand, which is about a simple option to restore previous output format.  It’s been a few months, people are either stuck with older pip-tools and can’t accept dependabot PRs anymore, or have to accept a PR with 100% diff.  As time passes and newer pip gets incompatible with older pip-tools, this won’t be tenable.
The last pip-tools version that generates compact requirements.txt annotations is `5.4.0`. I stick to using that.


@jdufresne 

Could you please at least weigh in on some of the proposed fixes? These include a structured output option, and two specific format options. I am *guessing* you would not support making the old style the default. I want to reduce wasted/undone work by getting input from the new-style "camp" written out here.
@AndydeCleyre sorry for the late response.

I like @cjerdonek's idea of adding a structured intermediate format that can then be manipulated further (outlined in #1377). pip-tools could ship with two formatters one with the multi-line annotations and one with the compact annotations. The project could choose one or the other as the default output. I personally like the multi-line as the default, but recognize others may prefer the compact.

I won't have the time to implement this, so it would be a welcome contribution from other maintainers or contributors. Thanks all for the feedback.

Resolving patch:
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -123,6 +123,12 @@ def _get_default_option(option_name: str) -> Any:
     default=True,
     help="Annotate results, indicating where dependencies come from",
 )
+@click.option(
+    "--annotation-style",
+    type=click.Choice(("line", "split")),
+    default="split",
+    help="Choose the format of annotation comments",
+)
 @click.option(
     "-U",
     "--upgrade/--no-upgrade",
@@ -244,6 +250,7 @@ def cli(
     header: bool,
     emit_trusted_host: bool,
     annotate: bool,
+    annotation_style: str,
     upgrade: bool,
     upgrade_packages: Tuple[str, ...],
     output_file: Union[LazyFile, IO[Any], None],
@@ -471,6 +478,7 @@ def cli(
         emit_index_url=emit_index_url,
         emit_trusted_host=emit_trusted_host,
         annotate=annotate,
+        annotation_style=annotation_style,
         strip_extras=strip_extras,
         generate_hashes=generate_hashes,
         default_index_url=repository.DEFAULT_INDEX_URL,
diff --git a/piptools/writer.py b/piptools/writer.py
--- a/piptools/writer.py
+++ b/piptools/writer.py
@@ -51,6 +51,23 @@ def _comes_from_as_string(ireq: InstallRequirement) -> str:
     return key_from_ireq(ireq.comes_from)
 
 
+def annotation_style_split(required_by: Set[str]) -> str:
+    sorted_required_by = sorted(required_by)
+    if len(sorted_required_by) == 1:
+        source = sorted_required_by[0]
+        annotation = "# via " + source
+    else:
+        annotation_lines = ["# via"]
+        for source in sorted_required_by:
+            annotation_lines.append("    #   " + source)
+        annotation = "\n".join(annotation_lines)
+    return annotation
+
+
+def annotation_style_line(required_by: Set[str]) -> str:
+    return f"# via {', '.join(sorted(required_by))}"
+
+
 class OutputWriter:
     def __init__(
         self,
@@ -61,6 +78,7 @@ def __init__(
         emit_index_url: bool,
         emit_trusted_host: bool,
         annotate: bool,
+        annotation_style: str,
         strip_extras: bool,
         generate_hashes: bool,
         default_index_url: str,
@@ -79,6 +97,7 @@ def __init__(
         self.emit_index_url = emit_index_url
         self.emit_trusted_host = emit_trusted_host
         self.annotate = annotate
+        self.annotation_style = annotation_style
         self.strip_extras = strip_extras
         self.generate_hashes = generate_hashes
         self.default_index_url = default_index_url
@@ -258,15 +277,16 @@ def _format_requirement(
             required_by.add(_comes_from_as_string(ireq))
 
         if required_by:
-            sorted_required_by = sorted(required_by)
-            if len(sorted_required_by) == 1:
-                source = sorted_required_by[0]
-                annotation = "    # via " + source
-            else:
-                annotation_lines = ["    # via"]
-                for source in sorted_required_by:
-                    annotation_lines.append("    #   " + source)
-                annotation = "\n".join(annotation_lines)
-            line = f"{line}\n{comment(annotation)}"
+            if self.annotation_style == "split":
+                annotation = annotation_style_split(required_by)
+                sep = "\n    "
+            elif self.annotation_style == "line":
+                annotation = annotation_style_line(required_by)
+                sep = "\n    " if ireq_hashes else "  "
+            else:  # pragma: no cover
+                raise ValueError("Invalid value for annotation style")
+            # 24 is one reasonable column size to use here, that we've used in the past
+            lines = f"{line:24}{sep}{comment(annotation)}".splitlines()
+            line = "\n".join(ln.rstrip() for ln in lines)
 
         return line


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -891,7 +891,7 @@ def test_generate_hashes_with_annotations(runner):
 
 
 @pytest.mark.network
-def test_generate_hashes_with_long_annotations(runner):
+def test_generate_hashes_with_split_style_annotations(runner):
     with open("requirements.in", "w") as fp:
         fp.write("Django==1.11.29\n")
         fp.write("django-debug-toolbar==1.11\n")
@@ -900,7 +900,7 @@ def test_generate_hashes_with_long_annotations(runner):
         fp.write("pytz==2020.4\n")
         fp.write("sqlparse==0.3.1\n")
 
-    out = runner.invoke(cli, ["--generate-hashes"])
+    out = runner.invoke(cli, ["--generate-hashes", "--annotation-style", "split"])
     assert out.stderr == dedent(
         f"""\
         #
@@ -946,6 +946,54 @@ def test_generate_hashes_with_long_annotations(runner):
     )
 
 
+@pytest.mark.network
+def test_generate_hashes_with_line_style_annotations(runner):
+    with open("requirements.in", "w") as fp:
+        fp.write("Django==1.11.29\n")
+        fp.write("django-debug-toolbar==1.11\n")
+        fp.write("django-storages==1.9.1\n")
+        fp.write("django-taggit==0.24.0\n")
+        fp.write("pytz==2020.4\n")
+        fp.write("sqlparse==0.3.1\n")
+
+    out = runner.invoke(cli, ["--generate-hashes", "--annotation-style", "line"])
+    assert out.stderr == dedent(
+        f"""\
+        #
+        # This file is autogenerated by pip-compile with python \
+{sys.version_info.major}.{sys.version_info.minor}
+        # To update, run:
+        #
+        #    pip-compile --annotation-style=line --generate-hashes
+        #
+        django==1.11.29 \\
+            --hash=sha256:014e3392058d94f40569206a24523ce254d55ad2f9f46c6550b0fe2e4f94cf3f \\
+            --hash=sha256:4200aefb6678019a0acf0005cd14cfce3a5e6b9b90d06145fcdd2e474ad4329c
+            # via -r requirements.in, django-debug-toolbar, django-storages, django-taggit
+        django-debug-toolbar==1.11 \\
+            --hash=sha256:89d75b60c65db363fb24688d977e5fbf0e73386c67acf562d278402a10fc3736 \\
+            --hash=sha256:c2b0134119a624f4ac9398b44f8e28a01c7686ac350a12a74793f3dd57a9eea0
+            # via -r requirements.in
+        django-storages==1.9.1 \\
+            --hash=sha256:3103991c2ee8cef8a2ff096709973ffe7106183d211a79f22cf855f33533d924 \\
+            --hash=sha256:a59e9923cbce7068792f75344ed7727021ee4ac20f227cf17297d0d03d141e91
+            # via -r requirements.in
+        django-taggit==0.24.0 \\
+            --hash=sha256:710b4d15ec1996550cc68a0abbc41903ca7d832540e52b1336e6858737e410d8 \\
+            --hash=sha256:bb8f27684814cd1414b2af75b857b5e26a40912631904038a7ecacd2bfafc3ac
+            # via -r requirements.in
+        pytz==2020.4 \\
+            --hash=sha256:3e6b7dd2d1e0a59084bcee14a17af60c5c562cdc16d828e8eba2e683d3a7e268 \\
+            --hash=sha256:5c55e189b682d420be27c6995ba6edce0c0a77dd67bfbe2ae6607134d5851ffd
+            # via -r requirements.in, django
+        sqlparse==0.3.1 \\
+            --hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \\
+            --hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548
+            # via -r requirements.in, django-debug-toolbar
+        """
+    )
+
+
 def test_filter_pip_markers(pip_conf, runner):
     """
     Check that pip-compile works with pip environment markers (PEP496)
@@ -1078,10 +1126,10 @@ def test_multiple_input_files_without_output_file(runner):
 
 
 @pytest.mark.parametrize(
-    ("option", "expected"),
+    ("options", "expected"),
     (
         pytest.param(
-            "--annotate",
+            ("--annotate",),
             f"""\
             #
             # This file is autogenerated by pip-compile with python \
@@ -1101,7 +1149,23 @@ def test_multiple_input_files_without_output_file(runner):
             id="annotate",
         ),
         pytest.param(
-            "--no-annotate",
+            ("--annotate", "--annotation-style", "line"),
+            f"""\
+            #
+            # This file is autogenerated by pip-compile with python \
+{sys.version_info.major}.{sys.version_info.minor}
+            # To update, run:
+            #
+            #    pip-compile --annotation-style=line --no-emit-find-links
+            #
+            small-fake-a==0.1         # via -c constraints.txt, small-fake-with-deps
+            small-fake-with-deps==0.1  # via -r requirements.in
+            Dry-run, so nothing updated.
+            """,
+            id="annotate line style",
+        ),
+        pytest.param(
+            ("--no-annotate",),
             f"""\
             #
             # This file is autogenerated by pip-compile with python \
@@ -1118,9 +1182,9 @@ def test_multiple_input_files_without_output_file(runner):
         ),
     ),
 )
-def test_annotate_option(pip_conf, runner, option, expected):
+def test_annotate_option(pip_conf, runner, options, expected):
     """
-    The output lines has have annotations if option is turned on.
+    The output lines have annotations if the option is turned on.
     """
     with open("constraints.txt", "w") as constraints_in:
         constraints_in.write("small-fake-a==0.1")
@@ -1128,7 +1192,7 @@ def test_annotate_option(pip_conf, runner, option, expected):
         req_in.write("-c constraints.txt\n")
         req_in.write("small_fake_with_deps")
 
-    out = runner.invoke(cli, [option, "-n", "--no-emit-find-links"])
+    out = runner.invoke(cli, [*options, "-n", "--no-emit-find-links"])
 
     assert out.stderr == dedent(expected)
     assert out.exit_code == 0
diff --git a/tests/test_writer.py b/tests/test_writer.py
--- a/tests/test_writer.py
+++ b/tests/test_writer.py
@@ -36,6 +36,7 @@ def writer(tmpdir_cwd):
             emit_index_url=True,
             emit_trusted_host=True,
             annotate=True,
+            annotation_style="split",
             generate_hashes=False,
             default_index_url=None,
             index_urls=[],
@@ -57,21 +58,21 @@ def test_format_requirement_annotation_editable(from_editable, writer):
 
     assert writer._format_requirement(
         ireq
-    ) == "-e git+git://fake.org/x/y.git#egg=y\n" + comment("    # via xyz")
+    ) == "-e git+git://fake.org/x/y.git#egg=y\n    " + comment("# via xyz")
 
 
 def test_format_requirement_annotation(from_line, writer):
     ireq = from_line("test==1.2")
     ireq.comes_from = "xyz"
 
-    assert writer._format_requirement(ireq) == "test==1.2\n" + comment("    # via xyz")
+    assert writer._format_requirement(ireq) == "test==1.2\n    " + comment("# via xyz")
 
 
 def test_format_requirement_annotation_lower_case(from_line, writer):
     ireq = from_line("Test==1.2")
     ireq.comes_from = "xyz"
 
-    assert writer._format_requirement(ireq) == "test==1.2\n" + comment("    # via xyz")
+    assert writer._format_requirement(ireq) == "test==1.2\n    " + comment("# via xyz")
 
 
 def test_format_requirement_for_primary(from_line, writer):
@@ -79,7 +80,7 @@ def test_format_requirement_for_primary(from_line, writer):
     ireq = from_line("test==1.2")
     ireq.comes_from = "xyz"
 
-    assert writer._format_requirement(ireq) == "test==1.2\n" + comment("    # via xyz")
+    assert writer._format_requirement(ireq) == "test==1.2\n    " + comment("# via xyz")
 
 
 def test_format_requirement_for_primary_lower_case(from_line, writer):
@@ -87,7 +88,7 @@ def test_format_requirement_for_primary_lower_case(from_line, writer):
     ireq = from_line("Test==1.2")
     ireq.comes_from = "xyz"
 
-    assert writer._format_requirement(ireq) == "test==1.2\n" + comment("    # via xyz")
+    assert writer._format_requirement(ireq) == "test==1.2\n    " + comment("# via xyz")
 
 
 def test_format_requirement_environment_marker(from_line, writer):