opened
philbinj
username_0
There seems to be an issue where globs produce different results when bazel is run from scratch (i.e. after a shutdown) vs when it is cached. Here is a python script that reproduces the issue:

```
import os, random, shutil, sys                                                                                                                                                                                                                                                                                                                          

NUM_FILES = 1000
NEW_PATH_PROB = 0.5
BASE = 'src/'

random.seed(42)


def generate_path(prefix):
    p = random.random()
    suffix = str(chr(ord('a') + random.randint(0, 25)))
    if p < NEW_PATH_PROB:
        return generate_path(os.path.join(prefix, suffix + '/'))
    return os.path.join(prefix, suffix)


def generate_paths():
    paths = []
    for i in range(NUM_FILES):
        paths.append(generate_path(BASE))
    paths = list(set(paths))
    return paths


def create_source_and_header(path):
    try:
        os.makedirs(os.path.dirname(path))
    except OSError, e:
        pass
    ccsrc = path + '.cc'
    hsrc = path + '.h'
    funcname = path.replace('/', '_')
    with open(ccsrc, 'w') as f:
        print >>f, """int %s() { return 1; }\n""" % funcname
    with open(hsrc, 'w') as f:
        print >>f, """int %s();\n""" % funcname


def create_build():
    buildname = 'BUILD'
    with open(buildname, 'w') as f:
        print >>f, """
cc_library(
    name = "src",
    srcs = glob(["src/**/*.cc"]),
    hdrs = glob(["src/**/*.h"]),
)
"""


def create_workspace():
    with open('WORKSPACE', 'w') as f:
        pass


def main():
    # Reset and rebuild local tree.
    try:
        shutil.rmtree(BASE)
    except OSError, e:
        pass
    paths = generate_paths()
    for p in paths:
        create_source_and_header(p)
    create_build()
    create_workspace()

    # Now start doing rebuilds.
    idx = 0
    os.system('bazel clean')
    while True:
        print '==== RUN %05d ====' % idx
        os.system('bazel shutdown')
        os.system('bazel build -c opt :src')
        with open('BUILD', 'a') as f:
            print >>f, '\n'
        os.system('bazel build -c opt :src')
        with open('BUILD', 'a') as f:
            print >>f, '\n'
        idx += 1


if __name__ == "__main__":
    main()
```

If you create a fresh directory and run the above script from within it a random tree of c++ files will be created with a corresponding BUILD and WORKSPACE. The script then alters the BUILD file and interlaces building after a shutdown with building when the server is running. With every invocation of `bazel build` all of the target c++ files are re-compiled from scratch despite the fact that neither the source files or the BUILD target ever changes.

If you alter the above script to always run `bazel shutdown` before build then no rebuilds are seen other than the first one. If you change the script to never run `bazel shutdown` then no rebuilds are seen other than the first one. So this is clearly some interaction between cached globs and the bazel server.

I believe this is high priority as it affects both correctness (bazel believes it needs to build when it shouldn't) and speed.
Unnecessary rebuilds observed when compiling targets with globs.
issue
created
damienmg
username_1
Sorry I added -s where it was usefull and never saw any rebuild, though there was loading invalidation which is expected. 

Replacing the main loop with that give me 0 rebuild (-s print no action):

```
    while idx < 10:
        print '==== RUN %05d ====' % idx
        os.system('bazel shutdown')
        os.system('bazel build -s -c opt :src' if idx > 0 else 'bazel build -c opt :src')
        with open('BUILD', 'a') as f:
            print >>f, '\n'
        os.system('bazel build -s -c opt :src')
        with open('BUILD', 'a') as f:
            print >>f, '\n'
        idx += 1
```
comment
closed
damienmg
username_1
issue
created
damienmg
username_1
No idea but we have been working hard a few months ago to fix invalidation issue with globs, could be it.
comment
created
philbinj
username_0
Got it, thank you!
comment