internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pantsbuild/pants

Issue:
when daemon enabled, cache dots / spaces are delayed
I ran into an issue where the cache progress dots were not being flushed to the terminal until an error happened, when I was running with the daemon enabled.

@stuhood said that it might be that with the daemon enabled, flushes only happen on newlines, which would explain the change in behavior.

Also, some of the dots showed up at the end of the run. :/


Discussion:
Starting this one.
Dropping this to focus on 5221 now that we have a plan.

Resolving patch:
diff --git a/src/python/pants/cache/artifact_cache.py b/src/python/pants/cache/artifact_cache.py
--- a/src/python/pants/cache/artifact_cache.py
+++ b/src/python/pants/cache/artifact_cache.py
@@ -161,6 +161,7 @@ def call_use_cached_files(tup):
       sys.stderr.write('.')
     else:
       sys.stderr.write(' ')
+    sys.stderr.flush()
     return res
   except NonfatalArtifactCacheError as e:
     logger.warn('Error calling use_cached_files in artifact cache: {0}'.format(e))


Test patch:
Repository: pantsbuild/pants

Issue:
Custom panic handler wipes out backtrace rendering
Inserting a panic in the native engine on master, you will see:
```
$ ./pants list 3rdparty/:
CRITICAL] panic at 'not yet implemented', src/rust/engine/src/scheduler.rs:40
CRITICAL] Please file a bug at https://github.com/pantsbuild/pants/issues.
fatal runtime error: failed to initiate panic, error 5
Fatal Python error: Aborted
```

...due to the custom panic handler installed by https://github.com/pantsbuild/pants/blob/7319b21030f020ffefabebda30e5f6fbf48cf248/src/python/pants/engine/native.py#L721

----

Passing `RUST_BACKTRACE=1` now disables the custom panic handler, but unfortunately this means that we won't get anything about the panic recorded to `pantsd.log`.

I think that we might need a middle ground whereby we:

1. add support to the custom handler for actually successfully rendering the panic to the log
2. either enable `RUST_BACKTRACE` by default, or override/ignore the flag in the custom handler


Discussion:

Resolving patch:
diff --git a/src/python/pants/engine/native.py b/src/python/pants/engine/native.py
--- a/src/python/pants/engine/native.py
+++ b/src/python/pants/engine/native.py
@@ -729,4 +729,7 @@ def tc(constraint):
     return self.gc(scheduler, self.lib.scheduler_destroy)
 
   def set_panic_handler(self):
-    self.lib.set_panic_handler()
+    if os.getenv("RUST_BACKTRACE", "0") != "1":
+      # The panic handler hides a lot of rust tracing which may be useful.
+      # Don't activate it when the user explicitly asks for rust backtraces.
+      self.lib.set_panic_handler()


Test patch:
Repository: pantsbuild/pants

Issue:
Nailgun processes that fail to be spawned don't explain why
If a nailgun process fails to start, the message that's presented is 

```
Failed to read nailgun output after 10.0 seconds!
```

The actual error message is usually found in the tools logs, which are in `.pants.d/ng/<tool-name>/{stdout,stderr}`.

A workaround that doesn't involve looking up the logs is to turn off nailgun. Then the message is presented directly.


Discussion:

Resolving patch:
diff --git a/src/python/pants/java/nailgun_executor.py b/src/python/pants/java/nailgun_executor.py
--- a/src/python/pants/java/nailgun_executor.py
+++ b/src/python/pants/java/nailgun_executor.py
@@ -21,7 +21,7 @@
 from pants.java.executor import Executor, SubprocessExecutor
 from pants.java.nailgun_client import NailgunClient
 from pants.pantsd.process_manager import FingerprintedProcessManager, ProcessGroup
-from pants.util.dirutil import safe_file_dump, safe_open
+from pants.util.dirutil import read_file, safe_file_dump, safe_open
 
 
 logger = logging.getLogger(__name__)
@@ -183,6 +183,7 @@ def _await_socket(self, timeout):
     """Blocks for the nailgun subprocess to bind and emit a listening port in the nailgun stdout."""
     with safe_open(self._ng_stdout, 'r') as ng_stdout:
       start_time = time.time()
+      accumulated_stdout = ''
       while 1:
         readable, _, _ = select.select([ng_stdout], [], [], self._SELECT_WAIT)
         if readable:
@@ -191,10 +192,18 @@ def _await_socket(self, timeout):
             return self._NG_PORT_REGEX.match(line).group(1)
           except AttributeError:
             pass
+          accumulated_stdout += line
 
         if (time.time() - start_time) > timeout:
+          stderr = read_file(self._ng_stderr)
           raise NailgunClient.NailgunError(
-            'Failed to read nailgun output after {sec} seconds!'.format(sec=timeout))
+            'Failed to read nailgun output after {sec} seconds!\n'
+            'Stdout:\n{stdout}\nStderr:\n{stderr}'.format(
+              sec=timeout,
+              stdout=accumulated_stdout,
+              stderr=stderr,
+            )
+          )
 
   def _create_ngclient(self, port, stdout, stderr, stdin):
     return NailgunClient(port=port, ins=stdin, out=stdout, err=stderr, workdir=get_buildroot())


Test patch:
Repository: pantsbuild/pants

Issue:
fmt then lint would fail because of scalafix
`./pants fmt <scala target>` will trigger `fmt.scalafmt` then `fmt.scalafix`. 

However the order may matter here, `scalafix` after `scalafmt` would rewrite the code again after reformatting. Therefore, when running `./pants lint ...` after, it will error at `lint.scalafmt` which can be confusing because we just `fmt` it.

so the their order should probably be swapped?

Found the issue while doing https://github.com/coursier/coursier/pull/877


Discussion:
Yes please.

Resolving patch:
diff --git a/src/python/pants/backend/jvm/register.py b/src/python/pants/backend/jvm/register.py
--- a/src/python/pants/backend/jvm/register.py
+++ b/src/python/pants/backend/jvm/register.py
@@ -216,8 +216,10 @@ def register_goals():
   task(name='jvm-dep-check', action=JvmDependencyCheck, serialize=False).install('lint')
 
   # Formatting.
-  task(name='scalafmt', action=ScalaFmtFormat, serialize=False).install('fmt')
+  # Scalafix has to go before scalafmt in order not to
+  # further change Scala files after scalafmt.
   task(name='scalafix', action=ScalaFixFix).install('fmt')
+  task(name='scalafmt', action=ScalaFmtFormat, serialize=False).install('fmt')
 
   # Running.
   task(name='jvm', action=JvmRun, serialize=False).install('run')


Test patch:
Repository: pantsbuild/pants

Issue:
contrib go plugin not able to recognize meta tag if meta ends with />
The regex only recognize `<meta xxxxxxxxxxxxx >` but not `<meta xxxxxxxxxx />`.


Discussion:

Resolving patch:
diff --git a/contrib/go/src/python/pants/contrib/go/subsystems/go_import_meta_tag_reader.py b/contrib/go/src/python/pants/contrib/go/subsystems/go_import_meta_tag_reader.py
--- a/contrib/go/src/python/pants/contrib/go/subsystems/go_import_meta_tag_reader.py
+++ b/contrib/go/src/python/pants/contrib/go/subsystems/go_import_meta_tag_reader.py
@@ -34,7 +34,7 @@ def register_options(cls, register):
           \s+
           content=['"](?P<root>[^\s]+)\s+(?P<vcs>[^\s]+)\s+(?P<url>[^\s]+)['"]
           \s*
-      >""", flags=re.VERBOSE)
+      /?>""", flags=re.VERBOSE)
 
   @classmethod
   def find_meta_tags(cls, page_html):


Test patch:
diff --git a/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_go_import_meta_tag_reader.py b/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_go_import_meta_tag_reader.py
--- a/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_go_import_meta_tag_reader.py
+++ b/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_go_import_meta_tag_reader.py
@@ -105,3 +105,24 @@ def test_no_meta_tag(self):
 
     meta_tag_content = GoImportMetaTagReader.find_meta_tags(test_html)
     self.assertEqual(meta_tag_content, [])
+
+  def test_meta_tag_end_with_forward_slash(self):
+    test_html = """
+    <!DOCTYPE html>
+    <html>
+    <head>
+    <meta name="go-import"
+          content="google.golang.org/notapi
+                   git
+                   https://code.googlesource.com/google-notapi-go-client" />
+    </head>
+    <body>
+    Nothing to see here.
+    Please <a href="https://godoc.org/google.golang.org/api/googleapi">move along</a>.
+    </body>
+    </html>
+    """
+
+    meta_tag_content = GoImportMetaTagReader.find_meta_tags(test_html)
+    self.assertEqual(meta_tag_content, [('google.golang.org/notapi', 'git',
+                      'https://code.googlesource.com/google-notapi-go-client')])
Repository: pantsbuild/pants

Issue:
Remove `root` argument to `pants.task.task.TaskBase._build_invalidator`
The argument is just there to support the now-deprecated `invalidate` task.


Discussion:

Resolving patch:
diff --git a/src/python/pants/task/task.py b/src/python/pants/task/task.py
--- a/src/python/pants/task/task.py
+++ b/src/python/pants/task/task.py
@@ -172,10 +172,9 @@ def __init__(self, *args, **kwargs):
     self._cache_factory = CacheSetup.create_cache_factory_for_task(self)
     self._force_invalidated = False
 
-  @memoized_method
-  def _build_invalidator(self, root=False):
-    build_task = None if root else self.fingerprint
-    return BuildInvalidator.Factory.create(build_task=build_task)
+  @memoized_property
+  def _build_invalidator(self):
+    return BuildInvalidator.Factory.create(build_task=self.fingerprint)
 
   def get_options(self):
     """Returns the option values for this task's scope.
@@ -285,7 +284,7 @@ def artifact_cache_writes_enabled(self):
 
   def invalidate(self):
     """Invalidates all targets for this task."""
-    self._build_invalidator().force_invalidate_all()
+    self._build_invalidator.force_invalidate_all()
 
   @property
   def create_target_dirs(self):
@@ -462,7 +461,7 @@ def _do_invalidation_check(self,
 
     cache_manager = InvalidationCacheManager(self.workdir,
                                              cache_key_generator,
-                                             self._build_invalidator(),
+                                             self._build_invalidator,
                                              invalidate_dependents,
                                              fingerprint_strategy=fingerprint_strategy,
                                              invalidation_report=self.context.invalidation_report,


Test patch:
Repository: pantsbuild/pants

Issue:
We display test successes even after an error that preempts test running
To reproduce:

Add a line above this one that raises some exception, so tests are never run: https://github.com/benjyw/pants/blob/039051735542d29ae02f4faa09c0c51c47292bf0/src/python/pants/backend/python/tasks/pytest_run.py#L521

Run some tests:

`./pants test.pytest tests/python/pants_test/init/::`

Output: 

```
<snip>
11:18:26 00:05     [pytest]
                   Invalidated 1 target.
                   scrubbed PYTHONPATH=/Users/benjyw/src/pants/src/python:/Users/benjyw/Dropbox/pycharm-debug/pycharm-debug.egg from py.test environment
11:18:26 00:05       [run]
                     
                   Failed to run test!
                   Traceback (most recent call last):
                     File "/Users/benjyw/src/pants/src/python/pants/backend/python/tasks/pytest_run.py", line 521, in _do_run_tests_with_args
                       raise Exception('OOPS')
                   Exception: OOPS
                   
                   tests/python/pants_test/init                                                    .....   SUCCESS
FAILURE


               Waiting for background workers to finish.
<snip>
```

Expected output: 

That, but without this line:
```
                   tests/python/pants_test/init                                                    .....   SUCCESS
```

Note that this happens even if you run without test caching!

`./pants test.pytest --cache-ignore tests/python/pants_test/init/::`



Discussion:
Looks like the bug is here:
https://github.com/benjyw/pants/blob/039051735542d29ae02f4faa09c0c51c47292bf0/src/python/pants/backend/python/tasks/pytest_run.py#L707

We need to claim all test targets are failed when there is no junit xml file to map failures from.
And, for posterity, this had nothing to do with caching.
OK, hopefully more clear:

![image](https://user-images.githubusercontent.com/157586/43978447-853d7984-9ca4-11e8-8e52-b538b77d7a1f.png)

PR coming shortly.

Resolving patch:
diff --git a/src/python/pants/task/testrunner_task_mixin.py b/src/python/pants/task/testrunner_task_mixin.py
--- a/src/python/pants/task/testrunner_task_mixin.py
+++ b/src/python/pants/task/testrunner_task_mixin.py
@@ -461,8 +461,12 @@ def _execute(self, all_targets):
       for partition in sorted(results):
         rv = results[partition]
         failed_targets = set(rv.failed_targets)
+        pre_execution_error = not failed_targets and not rv.success
         for target in partition:
-          if target in failed_targets:
+          if pre_execution_error:
+            log = self.context.log.warn
+            result = 'NOT RUN'
+          elif target in failed_targets:
             log = self.context.log.error
             result = rv
           else:


Test patch:
Repository: pantsbuild/pants

Issue:
The python_requirements macro retains no handle to the requirements file.
This means changes in the requirements file do not result in changes in a live target graph (pantsd).


Discussion:
Probably the simplest thing to do is have each generated PythonRequirementLibrary depend on a Files target that owns the requirement file.

Resolving patch:
diff --git a/src/python/pants/backend/python/python_requirements.py b/src/python/pants/backend/python/python_requirements.py
--- a/src/python/pants/backend/python/python_requirements.py
+++ b/src/python/pants/backend/python/python_requirements.py
@@ -58,9 +58,16 @@ def __call__(self, requirements_relpath='requirements.txt'):
                   raise ValueError('Only 1 --find-links url is supported per requirements file')
                 repository = value
 
+    requirements_file_target_name = requirements_relpath
+    self._parse_context.create_object('files',
+                                      name=requirements_file_target_name,
+                                      sources=[requirements_relpath])
+    requirements_dep = ':{}'.format(requirements_file_target_name)
+
     for requirement in requirements:
       req = self._parse_context.create_object('python_requirement', requirement,
                                               repository=repository)
       self._parse_context.create_object('python_requirement_library',
                                         name=req.project_name,
-                                        requirements=[req])
+                                        requirements=[req],
+                                        dependencies=[requirements_dep])


Test patch:
diff --git a/tests/python/pants_test/engine/legacy/BUILD b/tests/python/pants_test/engine/legacy/BUILD
--- a/tests/python/pants_test/engine/legacy/BUILD
+++ b/tests/python/pants_test/engine/legacy/BUILD
@@ -122,7 +122,6 @@ python_tests(
   dependencies = [
     '3rdparty/python:future',
     '3rdparty/python:mock',
-    '3rdparty/python:requirements_files',
     'src/python/pants/bin',
     'src/python/pants/build_graph',
     'src/python/pants/engine/legacy:graph',
Repository: pantsbuild/pants

Issue:
`PantsRequirement` environment markers are broken for `1.10.0.dev5`
Until these two bits of `release.sh` are updated to unpin pants and its contribs from 2.7:
https://github.com/pantsbuild/pants/blob/1ec7de8edd46827fa63e15b5d8aede88ac42d5f2/build-support/bin/release.sh#L63-L68

https://github.com/pantsbuild/pants/blob/1ec7de8edd46827fa63e15b5d8aede88ac42d5f2/build-support/bin/release.sh#L214-L215

The `pants_requirement` macro must emit environment markers that constrain the requirement to 2.7 as well:
https://github.com/pantsbuild/pants/blob/1ec7de8edd46827fa63e15b5d8aede88ac42d5f2/src/python/pants/backend/python/pants_requirement.py#L53



Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/python/pants_requirement.py b/src/python/pants/backend/python/pants_requirement.py
--- a/src/python/pants/backend/python/pants_requirement.py
+++ b/src/python/pants/backend/python/pants_requirement.py
@@ -50,7 +50,9 @@ def __call__(self, name=None, dist=None):
                                       msg='The {} target only works for pantsbuild.pants '
                                           'distributions, given {}'.format(self.alias, dist))
 
-    env_marker = "python_version>='3.4' or (python_version>='2.7' and python_version<'3')"
+    # Update the environment marker in lockstep with other changes as described in
+    #   https://github.com/pantsbuild/pants/issues/6450
+    env_marker = "python_version>='2.7' and python_version<'3'"
 
     requirement = PythonRequirement(requirement="{key}=={version} ; {env_marker}"
                                     .format(key=dist,


Test patch:
diff --git a/tests/python/pants_test/backend/python/test_pants_requirement.py b/tests/python/pants_test/backend/python/test_pants_requirement.py
--- a/tests/python/pants_test/backend/python/test_pants_requirement.py
+++ b/tests/python/pants_test/backend/python/test_pants_requirement.py
@@ -33,16 +33,12 @@ def key(python_requirement):
 
     req = list(python_requirement_library.payload.requirements)[0]
     self.assertIsNotNone(req.requirement.marker)
-    self.assertTrue(req.requirement.marker.evaluate(),
-                    'pants_requirement() should always work in the current test environment')
 
     def evaluate_version(version):
       return req.requirement.marker.evaluate({'python_version': version})
 
-    allowed_versions = {'2.7', '3.4', '3.5', '3.6', '3.7'}
-
-    self.assertFalse(evaluate_version('2.6'))
-    self.assertTrue(all(evaluate_version(v) for v in allowed_versions))
+    self.assertTrue(evaluate_version('2.7'))
+    self.assertFalse(all(evaluate_version(v) for v in ('2.6', '3.4', '3.5', '3.6', '3.7')))
 
   def test_default_name(self):
     self.add_to_build_file('3rdparty/python/pants', 'pants_requirement()')
Repository: pantsbuild/pants

Issue:
Allow `compatibility` attribute to be passed through to generated python targets
To do this:

- Override ` _copy_target_attributes` in `ApacheThriftPyGen`  to include 'compatibility', so it looks like this:

```
  @property
  def _copy_target_attributes(self):
    """Propagate these attributes to the synthetic python_library() target."""
    return ['provides', 'tags', 'scope', 'compatibility']
```

- See https://github.com/pantsbuild/pants/blob/039051735542d29ae02f4faa09c0c51c47292bf0/contrib/jax_ws/src/python/pants/contrib/jax_ws/tasks/jax_ws_gen.py#L90 as an example




Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/codegen/thrift/python/apache_thrift_py_gen.py b/src/python/pants/backend/codegen/thrift/python/apache_thrift_py_gen.py
--- a/src/python/pants/backend/codegen/thrift/python/apache_thrift_py_gen.py
+++ b/src/python/pants/backend/codegen/thrift/python/apache_thrift_py_gen.py
@@ -51,6 +51,10 @@ def execute_codegen(self, target, target_workdir):
         # leave as-is.
         pass
 
+  @property
+  def _copy_target_attributes(self):
+    return super(ApacheThriftPyGen, self)._copy_target_attributes + ['compatibility']
+
   def ignore_dup(self, tgt1, tgt2, rel_src):
     # Thrift generates all the intermediate __init__.py files, and they shouldn't
     # count as dups.


Test patch:
diff --git a/tests/python/pants_test/backend/codegen/thrift/python/test_apache_thrift_py_gen.py b/tests/python/pants_test/backend/codegen/thrift/python/test_apache_thrift_py_gen.py
--- a/tests/python/pants_test/backend/codegen/thrift/python/test_apache_thrift_py_gen.py
+++ b/tests/python/pants_test/backend/codegen/thrift/python/test_apache_thrift_py_gen.py
@@ -169,3 +169,21 @@ def test_namespace_effective(self):
                                stderr=subprocess.PIPE)
     _, stderr = process.communicate()
     self.assertEqual(0, process.returncode, stderr)
+
+  def test_compatibility_passthrough(self):
+    py2_thrift_library = self.make_target(spec='src/thrift/com/foo:py2',
+                                          target_type=PythonThriftLibrary,
+                                          sources=[],
+                                          compatibility=['CPython>=2.7,<3'])
+    _, py2_synthetic_target = self.generate_single_thrift_target(py2_thrift_library)
+
+    self.assertEqual(py2_thrift_library.compatibility, py2_synthetic_target.compatibility)
+
+    py3_thrift_library = self.make_target(spec='src/thrift/com/foo:py3',
+                                          target_type=PythonThriftLibrary,
+                                          sources=[],
+                                          compatibility=['CPython>=3,<3.7'])
+    _, py3_synthetic_target = self.generate_single_thrift_target(py3_thrift_library)
+
+    self.assertEqual(py3_thrift_library.compatibility, py3_synthetic_target.compatibility)
+    self.assertNotEqual(py3_synthetic_target.compatibility, py2_synthetic_target.compatibility)
Repository: pantsbuild/pants

Issue:
Resolve tests issues with deprecated source / sources target args
From thread [here](https://github.com/pantsbuild/pants/pull/7332#discussion_r263626801) - where cleaning up target arguments that had been deprecated (in https://github.com/pantsbuild/pants/pull/7332/commits/c09f9cbab4a14477db1b14a7ac0295655fe36893) led to a significant amount of test failures.

It seems there are quite a few `BUILD` files within pants that need to be brought in line with expected target arguments.


Discussion:
Note: The problem here isn't BUILD files (either `source` or `sources` should work fine), but the actual rule implementations. I'll try to pick this up soon; realistically, probably in a couple of weeks :)

Resolving patch:
diff --git a/src/python/pants/build_graph/target.py b/src/python/pants/build_graph/target.py
--- a/src/python/pants/build_graph/target.py
+++ b/src/python/pants/build_graph/target.py
@@ -100,25 +100,24 @@ def register_options(cls, register):
                     'in BUILD files that are not yet available in the current version of pants.')
 
     @classmethod
-    def check(cls, target, kwargs, payload):
+    def check(cls, target, kwargs):
       """
       :API: public
       """
-      cls.global_instance().check_unknown(target, kwargs, payload)
+      cls.global_instance().check_unknown(target, kwargs)
 
-    def check_unknown(self, target, kwargs, payload):
+    def check_unknown(self, target, kwargs):
       """
       :API: public
       """
       ignore_params = set((self.get_options().ignored or {}).get(target.type_alias, ()))
+      # The source argument is automatically promoted to the sources argument, and rules should
+      # always use the sources argument; don't error if people used source in their BUILD file.
+      # There doesn't appear to be an easy way to error if a Target subclass explicitly takes
+      # `source` as a named kwarg; it would be nice to error rather than silently swallow the value,
+      # if there were such a way.
+      ignore_params.add('source')
       unknown_args = {arg: value for arg, value in kwargs.items() if arg not in ignore_params}
-      # TODO(#7357) Remove these checks once test issues are resolved.
-      if 'sources' in payload.as_dict():
-        if 'source' in unknown_args:
-          unknown_args.pop('source')
-        if 'sources' in unknown_args:
-          unknown_args.pop('sources')
-          kwargs.pop('sources')
       ignored_args = {arg: value for arg, value in kwargs.items() if arg in ignore_params}
       if ignored_args:
         logger.debug('{target} ignoring the unimplemented arguments: {args}'
@@ -338,7 +337,7 @@ def __init__(self, name, address, build_graph, type_alias=None, payload=None, ta
     self._cached_exports_addresses = None
     self._no_cache = no_cache
     if kwargs:
-      self.Arguments.check(self, kwargs, self.payload)
+      self.Arguments.check(self, kwargs)
 
   @property
   def scope(self):


Test patch: