Repository: mesonbuild/meson

Issue:
builddir not usable if new option is added
Create a project.

```
mkdir foo
cd foo
echo 'int main(){return 0;}' > main.cpp
meson init
meson build
ninja -C build
```

Update project with an option that breaks "meson setup" on your machine. (The real usecase is that you did `git pull`)

```
echo "option('bar_root', type: 'string', value: 'C:\\\\doesnotexist')" > meson_options.txt
echo "lib = meson.get_compiler('cpp').find_library('bar', dirs: get_option('bar_root'))" >> meson.build
```

ninja will of course fail to build.

```
$ ninja -C build
ninja: Entering directory `build'
[0/1] Regenerating build files.
The Meson build system
Version: 0.47.0.dev1
Source dir: /tmp/foo
Build dir: /tmp/foo/build
Build type: native build
Project name: foo
Native C++ compiler: ccache c++ (gcc 7.3.0 "c++ (Ubuntu 7.3.0-16ubuntu3) 7.3.0")
Build machine cpu family: x86_64
Build machine cpu: x86_64
meson.build:9:0: ERROR:  Search directory C:\doesnotexist is not an absolute path.
A full log can be found at /tmp/foo/build/meson-logs/meson-log.txt
FAILED: build.ninja
/usr/bin/python3 /home/niklas/.local/bin/meson.py --internal regenerate /tmp/foo /tmp/foo/build --backend ninja
ninja: error: rebuilding 'build.ninja': subcommand failed
```

It is not possible to set the correct new value:

```
$ meson configure build/ -Dbar_root=/opt/bar
Meson configurator encountered an error:
ERROR: Unknown option bar_root.
```

This problem would be less annoying if there was a simple way to figure out the original `meson build -D...` command that was run. #3542


Discussion:
I can confirm this issue, and it still happens with my PR #3528 that rework cmd line options.

The reason is mconf can only set options that are defined in `coredata.dat` file because it won't parse again `meson.build` and `meson_options.txt`. But `coredata.dat` is not created/updated until configure succeed, see the comment above `cdf = env.dump_coredata()` in https://github.com/mesonbuild/meson/blob/master/mesonbuild/mesonmain.py#L189.
What would be the acceptable fix here? Reading `meson_options.txt` early in `mconf` and update `coredata.dat` with missing variables?
Hmm, the option could be in a subproject and that cannot be known until it parsed the whole project. You can run `meson --internal regenerate -Dbar_root=/opt/bar build/`and it should work... except I tested with meson master and it doesn't work... but it works with my PR #3528.

We probably should make `regenerate` command public, I'm currently working on a regenerate command that wipe everything and start again with the stored command line for issue 3542. We could have a "soft" and a "hard" regenerate.
Shouldn't ninja be able to figure out that `meson_options.txt` has changed and run the appropriate command to "import" the new options?
The problem here, if I understand correctly, is it does import the new option but its default value doesn't work. And that ninja cannot know. User must provide a new value, but currently user has no way to set that new value without wiping everything. Or did I miss-understood?
No I think you are right. I'm not 100% sure on the internals of meson here, but as far as I understand it `ninja reconfigure` doesn't update `coredata.dat` because `meson` fails. And `meson conf` doesn't work because `coredata.dat` wasn't updated. Please correct me if I'm wrong. So, I was thinking that, if `ninja reconfigure` does two steps 1. parse `meson_options.txt` and update `coredata.dat` and then 2. try to setup the project. Then a subsequent `meson conf` would work because `coredata.dat` knows about the new variables. Does that make sense?
> Hmm, the option could be in a subproject and that cannot be known until it parsed the whole project.

The subproject dir is known after parsing only the first line of the meson file. Perhaps it is possible to do something clever. Maybe `meson conf` could see if it finds the setting in any `meson_options.txt` file that it can find and then simply accept it?

Resolving patch:
diff --git a/mesonbuild/mesonmain.py b/mesonbuild/mesonmain.py
--- a/mesonbuild/mesonmain.py
+++ b/mesonbuild/mesonmain.py
@@ -43,6 +43,10 @@ def create_parser():
                    help=argparse.SUPPRESS)
     p.add_argument('--fatal-meson-warnings', action='store_true', dest='fatal_warnings',
                    help='Make all Meson warnings fatal')
+    p.add_argument('--reconfigure', action='store_true',
+                   help='Set options and reconfigure the project. Useful when new ' +
+                        'options have been added to the project and the default value ' +
+                        'is not working.')
     p.add_argument('builddir', nargs='?', default=None)
     p.add_argument('sourcedir', nargs='?', default=None)
     return p
@@ -57,8 +61,10 @@ def wrapmodetype(string):
 
 class MesonApp:
 
-    def __init__(self, dir1, dir2, handshake, options):
-        (self.source_dir, self.build_dir) = self.validate_dirs(dir1, dir2, handshake)
+    def __init__(self, options):
+        (self.source_dir, self.build_dir) = self.validate_dirs(options.builddir,
+                                                               options.sourcedir,
+                                                               options.reconfigure)
         self.options = options
 
     def has_build_file(self, dirname):
@@ -66,6 +72,15 @@ def has_build_file(self, dirname):
         return os.path.exists(fname)
 
     def validate_core_dirs(self, dir1, dir2):
+        if dir1 is None:
+            if dir2 is None:
+                if not os.path.exists('meson.build') and os.path.exists('../meson.build'):
+                    dir2 = '..'
+                else:
+                    raise MesonException('Must specify at least one directory name.')
+            dir1 = os.getcwd()
+        if dir2 is None:
+            dir2 = os.getcwd()
         ndir1 = os.path.abspath(os.path.realpath(dir1))
         ndir2 = os.path.abspath(os.path.realpath(dir2))
         if not os.path.exists(ndir1):
@@ -86,21 +101,23 @@ def validate_core_dirs(self, dir1, dir2):
             return ndir2, ndir1
         raise MesonException('Neither directory contains a build file %s.' % environment.build_filename)
 
-    def validate_dirs(self, dir1, dir2, handshake):
+    def validate_dirs(self, dir1, dir2, reconfigure):
         (src_dir, build_dir) = self.validate_core_dirs(dir1, dir2)
         priv_dir = os.path.join(build_dir, 'meson-private/coredata.dat')
         if os.path.exists(priv_dir):
-            if not handshake:
-                print('Directory already configured, exiting Meson. Just run your build command\n'
-                      '(e.g. ninja) and Meson will regenerate as necessary. If ninja fails, run ninja\n'
-                      'reconfigure to force Meson to regenerate.\n'
+            if not reconfigure:
+                print('Directory already configured.\n'
+                      '\nJust run your build command (e.g. ninja) and Meson will regenerate as necessary.\n'
+                      'If ninja fails, run "ninja reconfigure" or "meson --reconfigure"\n'
+                      'to force Meson to regenerate.\n'
                       '\nIf build failures persist, manually wipe your build directory to clear any\n'
                       'stored system data.\n'
-                      '\nTo change option values, run meson configure instead.')
-                sys.exit(0)
+                      '\nTo change option values, run "meson configure" instead.')
+                sys.exit(1)
         else:
-            if handshake:
-                raise RuntimeError('Something went terribly wrong. Please file a bug.')
+            if reconfigure:
+                print('Directory does not contain a valid build tree:\n{}'.format(build_dir))
+                sys.exit(1)
         return src_dir, build_dir
 
     def check_pkgconfig_envvar(self, env):
@@ -317,7 +334,11 @@ def run(original_args, mainfile):
 
     # No special command? Do the basic setup/reconf.
     if len(args) >= 2 and args[0] == '--internal':
-        if args[1] != 'regenerate':
+        if args[1] == 'regenerate':
+            # Rewrite "meson --internal regenerate" command line to
+            # "meson --reconfigure"
+            args = ['--reconfigure'] + args[2:]
+        else:
             script = args[1]
             try:
                 sys.exit(run_script_command(args[1:]))
@@ -325,29 +346,14 @@ def run(original_args, mainfile):
                 mlog.error('\nError in {} helper script:'.format(script))
                 mlog.exception(e)
                 sys.exit(1)
-        args = args[2:]
-        handshake = True
-    else:
-        handshake = False
 
     parser = create_parser()
 
     args = mesonlib.expand_arguments(args)
     options = parser.parse_args(args)
     coredata.parse_cmd_line_options(options)
-    dir1 = options.builddir
-    dir2 = options.sourcedir
     try:
-        if dir1 is None:
-            if dir2 is None:
-                if not os.path.exists('meson.build') and os.path.exists('../meson.build'):
-                    dir2 = '..'
-                else:
-                    raise MesonException('Must specify at least one directory name.')
-            dir1 = os.getcwd()
-        if dir2 is None:
-            dir2 = os.getcwd()
-        app = MesonApp(dir1, dir2, handshake, options)
+        app = MesonApp(options)
     except Exception as e:
         # Log directory does not exist, so just print
         # to stdout.


Test patch: