Repository: pantsbuild/pants

Issue #10455: Always use generated subtargets
Created: 2020-07-26T18:07:13Z
Currently, if there are multiple owners of a source file, then we do not use generated subtargets:

  
    
      pants/src/python/pants/engine/internals/graph.py
    
    
        Lines 277 to 284
      in
      0a61cf8
    
  
  
    

        
          
           if file_name in file_name_to_generated_address: 
        

        
          
               file_names_with_multiple_owners.add(file_name) 
        

        
          
               original_addresses_due_to_multiple_owners.add(candidate_tgt.address) 
        

        
          
               # We also add the original target of the generated address already stored. 
        

        
          
               already_stored_generated_address = file_name_to_generated_address[file_name] 
        

        
          
               original_addresses_due_to_multiple_owners.add( 
        

        
          
                   already_stored_generated_address.maybe_convert_to_base_target() 
        

        
          
               ) 
        
    
  


We solely do this to avoid the representation problem of how to represent a file address when there are multiple of them, but they have different generated_base_target_name. In Pants output, like ./pants list, we would show the same file foo.txt multiple times, which looks redudant, even though they're different targets. We'd need a syntax like foo.txt@:original to show the base target when there was ambiguity.
--
Why change this?

It makes the experience with file arguments more consistent for end users. We no longer have a special case. When using file args, you always get generated subtargets / files, no matter what.
We can remove AddressWithOrigin et al. The only reason we use WithOrigin right now is for SpecifiedSourceFiles. If we fix this issue, then there is no need for SpecifiedSourceFiles as described in #10454. This results in much less complexity for plugin authors to no longer need to know about OriginSpec.

--
Note that we will continue to need to use the original owning address if we detect a deleted file:

  
    
      pants/src/python/pants/engine/internals/graph.py
    
    
        Lines 259 to 261
      in
      0a61cf8
    
  
  
    

        
          
           deleted_files_matched = bool(set(matching_files) - all_source_files) 
        

        
          
           if deleted_files_matched: 
        

        
          
               original_addresses_due_to_deleted_files.add(candidate_tgt.address) 
        
    
  


This is important for --changed-since to work properly.

Duplicate issue #10354: Teach `dependees` how to work with generated subtargets
Created: 2020-07-14T21:21:22Z
Right now, ./pants dependees will always return original addresses, rather than generated subtargets:
▶ ./pants dependees src/python/pants/util
build-support/migration-support:tests
pants-plugins/src/python/internal_backend/utilities:utilities
src/python/pants/auth:auth
src/python/pants/backend/native/config:config
src/python/pants/backend/native/subsystems:subsystems
src/python/pants/backend/native/subsystems/utils:utils
src/python/pants/backend/pants_info:pants_info
src/python/pants/backend/project_info:project_info
src/python/pants/backend/python:python
src/python/pants/backend/python/dependency_inference:dependency_inference
src/python/pants/backend/python/dependency_inference:tests
src/python/pants/backend/python/lint/bandit:bandit
src/python/pants/backend/python/lint/black:black
src/python/pants/backend/python/lint/docformatter:docformatter
src/python/pants/backend/python/lint/flake8:flake8
src/python/pants/backend/python/lint/isort:isort
src/python/pants/backend/python/lint/pylint:pylint
src/python/pants/backend/python/rules:integration
src/python/pants/backend/python/rules:rules
src/python/pants/backend/python/rules:tests
src/python/pants/backend/python/subsystems:subsystems
src/python/pants/backend/python/typecheck/mypy:mypy

In contrast, dependencies understands generated subtargets:
▶ ./pants dependencies src/python/pants/backend/python/lint/isort:integration
src/python/pants/backend/python/lint/isort/rules.py
src/python/pants/backend/python/target_types.py
src/python/pants/base/specs.py
src/python/pants/core/goals/fmt.py
src/python/pants/core/goals/lint.py
src/python/pants/core/util_rules/determine_source_files.py
src/python/pants/engine/addresses.py
src/python/pants/engine/fs.py
src/python/pants/engine/rules.py
src/python/pants/engine/selectors.py
src/python/pants/engine/target.py
src/python/pants/testutil/external_tool_test_base.py
src/python/pants/testutil/option/util.py

It would be helpful if dependees also understood generated subtargets. The output would be more consistent with the rest of Pants and it would result in finer grained caching when using --changed-dependees.
The tricky part is that the goal must continue to understand traditional targets that are not generated.

Closing comment (2020-07-27T20:13:34Z): Partial dupe of #10455 I think, but will keep both open and linked to ensure that both are resolved by the changes.