Repository: pantsbuild/pants

Issue:
cache keys not matching cross-platform with v2 engine
I'm running pants 1.3.0rc2.  As far as I know, this was an issue on rc1 and rc0 as well.  We have two continuous integration machines (Centos 7.3.1611) that run the build with every commit.  Our developer machines are running osx.  I started noticing that the developer machines were not finding any hits from the cache created on the build machines.  All zinc entries would miss.  Also, entries for one of our custom tasks would miss.  The debug logs showed that the requests to the cache were for keys that didn't exist in the cache and 404s were returned.

The cache did work between the two build machines.  One of the build machines would find hits for artifacts created on the other machine and vice-versa.  It was only the developer machines missing.

I first did a comparison of options between the build options on the build machines and developer machine.  I found no significant differences, only things like python path.  I also used the same pants.ini file on the developer machine and still got misses.

I wrote a task to dump out `target.invalidation_hash(None)` on a small target and ran this on both machines.  This hash differs.  If I dump all fields of the payload of the target, I see that `field.sources.files_hash` differs for the `sources` field of the target.

I used python to compute the sha1 of the files and their paths.  These all match, but the `invalidation_hash` is still different.  On both machines the manually computed hash is different from `invalidation_hash`.

When I switch to the old engine, invalidation_hash matches on the machines, and also matches my manually computed version of the hash.  The cache works normally between machines (mostly hits) when I switch to the old engine.

I will attach the source of my task that dumps info about the targets and the output of this on both machines.


Discussion:
A small task that dumps out fingerprints/hashes of pieces of the target:

```python

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
                        unicode_literals, with_statement)

from pants.base.exceptions import TaskError
from pants.task.task import Task
from pants.backend.jvm.tasks.jvm_compile.jvm_compile import ResolvedJarAwareFingerprintStrategy
from pants.source.payload_fields import SourcesField

import os
from hashlib import sha1
from pants.base.build_environment import get_buildroot


class FingerTask(Task):
    # don't cache, as we don't generate artifacts
    @property
    def cache_target_dirs(self):
        return False

    def execute(self):
        for target in self.context.targets():
            print("target: " + target.name)
            print("  invalidation: " + target.invalidation_hash(None))
            for key in target.payload._fields:
                field = target.payload.get_field(key)
                val = target.payload.get_field_value(key)
                if field:
                    print("  " + key + " -> " + repr(val) + ": " + field.fingerprint())
                else:
                    print("  " + key + " -> " + repr(val))
                if isinstance(field, SourcesField):
                    print("    rel_path: " + field.rel_path)
                    print("    sources: " + repr(field.sources))
                    print("    files_hash: " + repr(field.sources.files_hash))
                    h1 = sha1()
                    for path in sorted(field.sources.files):
                        file_digest = None
                        h = sha1()
                        h.update(path)
                        h1.update(path)
                        name_digest = h.hexdigest()
                        with open(os.path.join(get_buildroot(), field.sources.rel_root, path), 'rb') as f:
                            h = sha1()
                            data = f.read()
                            h.update(data)
                            h1.update(data)
                            file_digest = h.hexdigest()
                        print("      " + path + " " + name_digest + " " + file_digest)
                    print("      computed: " + repr(h1.digest()))
```

Output of the above target, run against a target that has a handful of Java sources, run on the Mac laptop

```
  invalidation: d0b6b00f7d71204a294d19f9d23c572244698945-DefaultFingerprintStrategy
  excludes -> None
  exports -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  javac_plugin_args -> None: 2be88ca4242c76e8253ac62474851065032d6833
  scalac_plugin_args -> None: 2be88ca4242c76e8253ac62474851065032d6833
  resources -> []: 97d170e1550eee4afc0af065b78cda302a97674c
  scalac_plugins -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  sources -> <pants.source.payload_fields.SourcesField object at 0x1057b35d0>: 3088198e163b7afad40eff33afa43408dd33afd8
    rel_path: nucleus/java
    sources: EagerFilesetWithSpec(rel_root='nucleus/java', files=('com/r9/nucleus/message/MessageBundleMapBuilder.java', 'com/r9/nucleus/message/MessageHelperWrapper.java', 'com/r9/nucleus/message/MessageBundle.java', 'com/r9/nucleus/message/IMessageHelper.java', 'com/r9/nucleus/message/LocaleContext.java', 'com/r9/nucleus/message/MessageDebug.java', 'com/r9/nucleus/commons/Context.java', 'com/r9/nucleus/commons/ContextKeys.java', 'com/r9/nucleus/commons/ContextKey.java', 'com/r9/nucleus/commons/Singleton.java', 'com/r9/nucleus/commons/ResettableSingleton.java', 'com/r9/nucleus/commons/DisposableObject.java', 'com/r9/nucleus/commons/DisposableRegistry.java'))
    files_hash: 'P(\xe8\xfb\xc2\x06/\xa1\xab{[:\xec\x88"\xeb\xa9\x87\xed\xa6Y\t\xbb\xb6G8\xf7\xf3Q\x1f\xd0u'
      com/r9/nucleus/commons/Context.java a7ea1fc6426856c690c7b82397f4d6ca0955c90f b798e7b6769c145590e6309e2f33ff8cbd287a39
      com/r9/nucleus/commons/ContextKey.java 86adb383c7ed5b8375001fed23af97176d23ce2a c7faaf6b3d796051f4c9f469755f77bf380a78b5
      com/r9/nucleus/commons/ContextKeys.java ff814848409e6a537954348b239ddc8b90c8c70c 09fd81d0fc17167fc14fe00a775fba6618d2f7ac
      com/r9/nucleus/commons/DisposableObject.java cc06a981df3b7810629e61fe4d5f6c2443ce6eb5 16bdc816bbb5556e6a284d255f0ccd91c3fa90d6
      com/r9/nucleus/commons/DisposableRegistry.java ef0234b398ff09e91c36da1cfe6907098cb79f94 0a35e16b92955c015ddd6fd4e42abc561ef1159a
      com/r9/nucleus/commons/ResettableSingleton.java 59ba1e2dc94070a0876491188175e1b880fb7929 a6f283564cb23f4aba6b2dff44d8eb2f2a255ac4
      com/r9/nucleus/commons/Singleton.java 2aa1df15dc45ec772d12ae60d59edc385df0ad4a bca206d4ac92330a58fa7c4a32239a713bfb1ed2
      com/r9/nucleus/message/IMessageHelper.java 36b4c7bd3e9ff09e7814bda3b5bfc0fa6869375d 1c4eda6611fb7e73e9ec6c0e0fd457ca6dba0ca3
      com/r9/nucleus/message/LocaleContext.java 00c51763144c315b529be47746009cacb1abc881 6a11e7385791eaefad2bd7a6c5da54e752fc94ba
      com/r9/nucleus/message/MessageBundle.java 0274cddda3c997c8e366002df208a35c26b4495a f33ab3cb21c7e8f1d669f918d28122edf8ad5e33
      com/r9/nucleus/message/MessageBundleMapBuilder.java 3ea067201759e4d5c84b7d4f8619caae852811d6 c5d760075121e8fc8ba82cb65e3c6d59193fc73f
      com/r9/nucleus/message/MessageDebug.java c8c3f6503634bdf864c6a8f84f28e09753ddb271 baf244c395da7a472fac5dacfda6a000f66ed723
      com/r9/nucleus/message/MessageHelperWrapper.java 68de0975bfa5fa642c9b234cb09dd30cc7fc538e af090fae44c6cf2de329b045651c0ab8bcc615d7
      computed: '\xe6\xc7u\xd1G\xee\xf38\\\x166\xb3\xe4\x8e\x1f\xc8\xcc\xc2P\xd8'
  javac_plugins -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  fatal_warnings -> None: 2be88ca4242c76e8253ac62474851065032d6833
  platform -> 'java8': 9580dfe515928b0cd8bae7985d5575d7069e7b37
  provides -> None
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  strict_deps -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  zinc_file_manager -> None: 2be88ca4242c76e8253ac62474851065032d6833
target: guava
  invalidation: f2134f6a5e9bb01a00b300930edc182691a80095-DefaultFingerprintStrategy
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  excludes -> None
  jars -> (JarDependency(org='com.google.guava', base_name='guava', rev='20.0', force=False, ext=None, url=None, apidocs=None, classifier=None, mutable=None, intransitive=True, excludes=(), base_path='alien/3rdparty'),): 41bffc56836cc257a3ee1d6a2e0dd56901b3cbfd
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  managed_dependencies -> None: 2be88ca4242c76e8253ac62474851065032d6833
target: ehcache-unstable-compile-4d4a31e32d8a8fc16067c06e473a90d8b9e5c4e4
  invalidation: 81d96d117e19621d1248290957a8b01baf8001a0-DefaultFingerprintStrategy
  scope_string -> 'compile': 146813bbe7cd50ec54846702ceefc82501b5551a
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
target: ehcache
  invalidation: a7954a0d2af2cdfa0defb2a2c0bb037f881ec574-DefaultFingerprintStrategy
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  excludes -> None
  jars -> (JarDependency(org='net.sf.ehcache', base_name='ehcache', rev='2.8.5', force=False, ext=None, url=None, apidocs=None, classifier=None, mutable=None, intransitive=True, excludes=(), base_path='alien/3rdparty'),): 925ddeda11d2f4685a9529911c8a51e3cae526cf
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  managed_dependencies -> None: 2be88ca4242c76e8253ac62474851065032d6833
target: spring-core
  invalidation: c685034613c3534dfc106389ce401d01454e5cc7-DefaultFingerprintStrategy
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  excludes -> None
  jars -> (JarDependency(org='org.springframework', base_name='spring-core', rev='4.3.3.RELEASE', force=False, ext=None, url=None, apidocs=None, classifier=None, mutable=None, intransitive=True, excludes=(), base_path='alien/3rdparty'),): 0480cb4e8cd810fa5a50e5c801f35b018da58384
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  managed_dependencies -> None: 2be88ca4242c76e8253ac62474851065032d6833
```

Output from the linux build machine on the same target:

```
  invalidation: 0a63ec037e5e135f890761cb1ad5ff7de78e2b0e-DefaultFingerprintStrategy
  excludes -> None
  exports -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  javac_plugin_args -> None: 2be88ca4242c76e8253ac62474851065032d6833
  scalac_plugin_args -> None: 2be88ca4242c76e8253ac62474851065032d6833
  resources -> []: 97d170e1550eee4afc0af065b78cda302a97674c
  scalac_plugins -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  sources -> <pants.source.payload_fields.SourcesField object at 0x2c9bb10>: e9ed77f7dc4eb8ce21241b0f26bbb81ddce879c7
    rel_path: nucleus/java
    sources: EagerFilesetWithSpec(rel_root='nucleus/java', files=('com/r9/nucleus/message/MessageBundleMapBuilder.java', 'com/r9/nucleus/message/MessageHelperWrapper.java', 'com/r9/nucleus/message/MessageBundle.java', 'com/r9/nucleus/message/IMessageHelper.java', 'com/r9/nucleus/message/LocaleContext.java', 'com/r9/nucleus/message/MessageDebug.java', 'com/r9/nucleus/commons/Context.java', 'com/r9/nucleus/commons/ContextKeys.java', 'com/r9/nucleus/commons/ContextKey.java', 'com/r9/nucleus/commons/Singleton.java', 'com/r9/nucleus/commons/ResettableSingleton.java', 'com/r9/nucleus/commons/DisposableObject.java', 'com/r9/nucleus/commons/DisposableRegistry.java'))
    files_hash: 'NaV\xc8\x08A\xdf8\x9f\x1e\xc3\xe7]\xa3\x1b\xbb\xe0#\x92\x81\xae\xaez\x17\xfc8\x89\x16\xcd\xb1\xe0\x8d'
      com/r9/nucleus/commons/Context.java a7ea1fc6426856c690c7b82397f4d6ca0955c90f b798e7b6769c145590e6309e2f33ff8cbd287a39
      com/r9/nucleus/commons/ContextKey.java 86adb383c7ed5b8375001fed23af97176d23ce2a c7faaf6b3d796051f4c9f469755f77bf380a78b5
      com/r9/nucleus/commons/ContextKeys.java ff814848409e6a537954348b239ddc8b90c8c70c 09fd81d0fc17167fc14fe00a775fba6618d2f7ac
      com/r9/nucleus/commons/DisposableObject.java cc06a981df3b7810629e61fe4d5f6c2443ce6eb5 16bdc816bbb5556e6a284d255f0ccd91c3fa90d6
      com/r9/nucleus/commons/DisposableRegistry.java ef0234b398ff09e91c36da1cfe6907098cb79f94 0a35e16b92955c015ddd6fd4e42abc561ef1159a
      com/r9/nucleus/commons/ResettableSingleton.java 59ba1e2dc94070a0876491188175e1b880fb7929 a6f283564cb23f4aba6b2dff44d8eb2f2a255ac4
      com/r9/nucleus/commons/Singleton.java 2aa1df15dc45ec772d12ae60d59edc385df0ad4a bca206d4ac92330a58fa7c4a32239a713bfb1ed2
      com/r9/nucleus/message/IMessageHelper.java 36b4c7bd3e9ff09e7814bda3b5bfc0fa6869375d 1c4eda6611fb7e73e9ec6c0e0fd457ca6dba0ca3
      com/r9/nucleus/message/LocaleContext.java 00c51763144c315b529be47746009cacb1abc881 6a11e7385791eaefad2bd7a6c5da54e752fc94ba
      com/r9/nucleus/message/MessageBundle.java 0274cddda3c997c8e366002df208a35c26b4495a f33ab3cb21c7e8f1d669f918d28122edf8ad5e33
      com/r9/nucleus/message/MessageBundleMapBuilder.java 3ea067201759e4d5c84b7d4f8619caae852811d6 c5d760075121e8fc8ba82cb65e3c6d59193fc73f
      com/r9/nucleus/message/MessageDebug.java c8c3f6503634bdf864c6a8f84f28e09753ddb271 baf244c395da7a472fac5dacfda6a000f66ed723
      com/r9/nucleus/message/MessageHelperWrapper.java 68de0975bfa5fa642c9b234cb09dd30cc7fc538e af090fae44c6cf2de329b045651c0ab8bcc615d7
      computed: '\xe6\xc7u\xd1G\xee\xf38\\\x166\xb3\xe4\x8e\x1f\xc8\xcc\xc2P\xd8'
  javac_plugins -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  fatal_warnings -> None: 2be88ca4242c76e8253ac62474851065032d6833
  platform -> 'java8': 9580dfe515928b0cd8bae7985d5575d7069e7b37
  provides -> None
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  strict_deps -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  zinc_file_manager -> None: 2be88ca4242c76e8253ac62474851065032d6833
```

The `files_hash` values differ, but the computed sha1 values of the files are the same.

The output when `enable_v2_engine: False` is set as a GLOBAL option:

```
  invalidation: a433b10a98f122a73c9f4291cab3b3f085931f45-DefaultFingerprintStrategy
  excludes -> None
  exports -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  javac_plugin_args -> None: 2be88ca4242c76e8253ac62474851065032d6833
  scalac_plugin_args -> None: 2be88ca4242c76e8253ac62474851065032d6833
  resources -> []: 97d170e1550eee4afc0af065b78cda302a97674c
  scalac_plugins -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  sources -> <pants.source.payload_fields.SourcesField object at 0x10f17d5d0>: 7201f15de484f6d57970e432554448ad9f16f50e
    rel_path: nucleus/java
    sources: <pants.source.wrapped_globs.LazyFilesetWithSpec object at 0x11071ac90>
    files_hash: '\xe6\xc7u\xd1G\xee\xf38\\\x166\xb3\xe4\x8e\x1f\xc8\xcc\xc2P\xd8'
      com/r9/nucleus/commons/Context.java a7ea1fc6426856c690c7b82397f4d6ca0955c90f b798e7b6769c145590e6309e2f33ff8cbd287a39
      com/r9/nucleus/commons/ContextKey.java 86adb383c7ed5b8375001fed23af97176d23ce2a c7faaf6b3d796051f4c9f469755f77bf380a78b5
      com/r9/nucleus/commons/ContextKeys.java ff814848409e6a537954348b239ddc8b90c8c70c 09fd81d0fc17167fc14fe00a775fba6618d2f7ac
      com/r9/nucleus/commons/DisposableObject.java cc06a981df3b7810629e61fe4d5f6c2443ce6eb5 16bdc816bbb5556e6a284d255f0ccd91c3fa90d6
      com/r9/nucleus/commons/DisposableRegistry.java ef0234b398ff09e91c36da1cfe6907098cb79f94 0a35e16b92955c015ddd6fd4e42abc561ef1159a
      com/r9/nucleus/commons/ResettableSingleton.java 59ba1e2dc94070a0876491188175e1b880fb7929 a6f283564cb23f4aba6b2dff44d8eb2f2a255ac4
      com/r9/nucleus/commons/Singleton.java 2aa1df15dc45ec772d12ae60d59edc385df0ad4a bca206d4ac92330a58fa7c4a32239a713bfb1ed2
      com/r9/nucleus/message/IMessageHelper.java 36b4c7bd3e9ff09e7814bda3b5bfc0fa6869375d 1c4eda6611fb7e73e9ec6c0e0fd457ca6dba0ca3
      com/r9/nucleus/message/LocaleContext.java 00c51763144c315b529be47746009cacb1abc881 6a11e7385791eaefad2bd7a6c5da54e752fc94ba
      com/r9/nucleus/message/MessageBundle.java 0274cddda3c997c8e366002df208a35c26b4495a f33ab3cb21c7e8f1d669f918d28122edf8ad5e33
      com/r9/nucleus/message/MessageBundleMapBuilder.java 3ea067201759e4d5c84b7d4f8619caae852811d6 c5d760075121e8fc8ba82cb65e3c6d59193fc73f
      com/r9/nucleus/message/MessageDebug.java c8c3f6503634bdf864c6a8f84f28e09753ddb271 baf244c395da7a472fac5dacfda6a000f66ed723
      com/r9/nucleus/message/MessageHelperWrapper.java 68de0975bfa5fa642c9b234cb09dd30cc7fc538e af090fae44c6cf2de329b045651c0ab8bcc615d7
      computed: '\xe6\xc7u\xd1G\xee\xf38\\\x166\xb3\xe4\x8e\x1f\xc8\xcc\xc2P\xd8'
  javac_plugins -> (): 97d170e1550eee4afc0af065b78cda302a97674c
  fatal_warnings -> None: 2be88ca4242c76e8253ac62474851065032d6833
  platform -> 'java8': 9580dfe515928b0cd8bae7985d5575d7069e7b37
  provides -> None
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  strict_deps -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  zinc_file_manager -> None: 2be88ca4242c76e8253ac62474851065032d6833
target: guava
  invalidation: f2134f6a5e9bb01a00b300930edc182691a80095-DefaultFingerprintStrategy
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  excludes -> None
  jars -> (JarDependency(org='com.google.guava', base_name='guava', rev='20.0', force=False, ext=None, url=None, apidocs=None, classifier=None, mutable=None, intransitive=True, excludes=(), base_path=u'alien/3rdparty'),): 41bffc56836cc257a3ee1d6a2e0dd56901b3cbfd
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  managed_dependencies -> None: 2be88ca4242c76e8253ac62474851065032d6833
target: ehcache-unstable-compile-4d4a31e32d8a8fc16067c06e473a90d8b9e5c4e4
  invalidation: 81d96d117e19621d1248290957a8b01baf8001a0-DefaultFingerprintStrategy
  scope_string -> 'compile': 146813bbe7cd50ec54846702ceefc82501b5551a
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
target: ehcache
  invalidation: a7954a0d2af2cdfa0defb2a2c0bb037f881ec574-DefaultFingerprintStrategy
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  excludes -> None
  jars -> (JarDependency(org='net.sf.ehcache', base_name='ehcache', rev='2.8.5', force=False, ext=None, url=None, apidocs=None, classifier=None, mutable=None, intransitive=True, excludes=(), base_path=u'alien/3rdparty'),): 925ddeda11d2f4685a9529911c8a51e3cae526cf
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  managed_dependencies -> None: 2be88ca4242c76e8253ac62474851065032d6833
target: spring-core
  invalidation: c685034613c3534dfc106389ce401d01454e5cc7-DefaultFingerprintStrategy
  scope_string -> 'None': 44cde2d98f429b8f45778a18c62db43ecb9bb564
  excludes -> None
  jars -> (JarDependency(org='org.springframework', base_name='spring-core', rev='4.3.3.RELEASE', force=False, ext=None, url=None, apidocs=None, classifier=None, mutable=None, intransitive=True, excludes=(), base_path=u'alien/3rdparty'),): 0480cb4e8cd810fa5a50e5c801f35b018da58384
  transitive -> True: 5ffe533b830f08a0326348a9160afafc8ada44db
  managed_dependencies -> None: 2be88ca4242c76e8253ac62474851065032d6833
```

In the case, the `computed` hash and the `file_hash` are the same.  This output matches on the build machine and the Mac laptop.

Thanks for the report!

@youngderekm : On a hunch: what do you see for v2 if you remove the sorting here:
```
for path in sorted(field.sources.files):
```
?

I'll look into reproducing this.
@stuhood when I remove the sorting the files are listed in a different order on both machines: com/r9/nucleus/message files are listed before com/r9/nucleus/commons.  Both machines return the files in matching order. 
Great, thanks for the data. That is unexpected!

On May 22, 2017 5:49 AM, "Derek Young" <notifications@github.com> wrote:

> @stuhood <https://github.com/stuhood> when I remove the sorting the files
> are listed in a different order on both machines: com/r9/nucleus/message
> files are listed before com/r9/nucleus/commons. Both machines return the
> files in matching order.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pantsbuild/pants/issues/4610#issuecomment-303090351>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAC2lF_rr6z9UiS9HN73n6kMgNnrFwthks5r8YRHgaJpZM4Ng6dR>
> .
>


Resolving patch:
diff --git a/src/python/pants/engine/legacy/graph.py b/src/python/pants/engine/legacy/graph.py
--- a/src/python/pants/engine/legacy/graph.py
+++ b/src/python/pants/engine/legacy/graph.py
@@ -346,8 +346,6 @@ def _eager_fileset_with_spec(spec_path, filespec, snapshot):
     relpath_adjusted_filespec['exclude'] = [FilesetRelPathWrapper.to_filespec(e['globs'], spec_path)
                                             for e in filespec['exclude']]
 
-  # NB: In order to preserve declared ordering, we record a list of matched files
-  # independent of the file hash dict.
   return EagerFilesetWithSpec(spec_path,
                               relpath_adjusted_filespec,
                               files=files,


Test patch:
diff --git a/tests/python/pants_test/engine/legacy/test_graph.py b/tests/python/pants_test/engine/legacy/test_graph.py
--- a/tests/python/pants_test/engine/legacy/test_graph.py
+++ b/tests/python/pants_test/engine/legacy/test_graph.py
@@ -150,13 +150,18 @@ def test_invalidate_fsnode_incremental(self):
         node_count, last_node_count = scheduler.node_count(), node_count
         self.assertLess(node_count, last_node_count)
 
-  def test_sources_ordering(self):
-    spec = 'testprojects/src/resources/org/pantsbuild/testproject/ordering'
+  def _ordering_test(self, spec, expected_sources=None):
+    expected_sources = expected_sources or ['p', 'a', 'n', 't', 's', 'b', 'u', 'i', 'l', 'd']
     with self.open_scheduler([spec]) as (graph, _, _):
       target = graph.get_target(Address.parse(spec))
       sources = [os.path.basename(s) for s in target.sources_relative_to_buildroot()]
-      self.assertEquals(['p', 'a', 'n', 't', 's', 'b', 'u', 'i', 'l', 'd'],
-                        sources)
+      self.assertEquals(expected_sources, sources)
+
+  def test_sources_ordering_literal(self):
+    self._ordering_test('testprojects/src/resources/org/pantsbuild/testproject/ordering:literal')
+
+  def test_sources_ordering_glob(self):
+    self._ordering_test('testprojects/src/resources/org/pantsbuild/testproject/ordering:globs')
 
   def test_target_macro_override(self):
     """Tests that we can "wrap" an existing target type with additional functionality.