Repository: olofk/fusesoc

Issue:
How to use fusesoc.conf with RCS?
I've just started trying to use `fusesoc.conf` to manage libraries and am not sure I'm using it correctly.  My aim is to check in the configuration file such that when a user checks out a clean repo and uses `fusesoc`, all libraries that are missing get populated.

Running something like `fusesoc library add idex_generators https://github.com/idex-biometrics/fusesoc-generators` results in a configuration file as follows:

```
[library.idex_generators]
location = fusesoc_libraries/idex_generators
sync-uri = https://github.com/idex-biometrics/fusesoc-generators
sync-type = git
auto-sync = true
```

but if I commit that file and run `fusesoc` as normal I get a warning about:

```
WARNING: Failed to register library 'fusesoc_libraries/idex_generators is not a directory'
```

The `fusesoc library` sub-command does have an `update` argument but that seems to incorrectly identify the library as `local`:

```
$ fusesoc library update
WARNING: Failed to register library 'fusesoc_libraries/idex_generators is not a directory'
INFO: /home/shareefj/git/GN-651 : sync-type is local. Ignoring update
```

Perhaps that's the issue?  Is running `update` the correct way to populate the libraries but it's not working due to a bug in determining the `sync-type`?


Discussion:
I guess the main problem is that checking in fusesoc.conf is not an intended use-case. It does seem like it _almost_ works though, so let's try to make what you're trying to do possible.

Yes, that looks like a bug. There are some heuristics involved here and I'm frankly not super familiar with that part of the code as I didn't write it myself. I think it's a fair assumption that `fusesoc library update`, but I wonder if that INFO message is really related to the same library as was mentioned in the warning.

What I guess happens is that FuseSoC tries to go into the library and run `git update` and fails because the library doesn't exist. I think we could change that to instead do a fresh clone in that case, which would probably solve your use-case
So is the original use case that a user should always run `fusesoc library add` on each clean checkout?

I can take a closer look at the code and try and work out what's going on.
Library managment in FuseSoC could need some more love in general. The use-case you describe just wasn't considered by anyone. The assumption was that the workspace directory is a throw-away directory that isn't under version control. But I do see the benefit of checking in a base workspace dir, especially if you want to put some utilities and stuff inside that.



Patch:
diff --git a/fusesoc/fusesoc.py b/fusesoc/fusesoc.py
index 37cc25e3..5f163fc1 100644
--- a/fusesoc/fusesoc.py
+++ b/fusesoc/fusesoc.py
@@ -37,8 +37,16 @@ def _register_libraries(self):
             try:
                 self.add_library(library)
             except (RuntimeError, OSError) as e:
-                _s = "Failed to register library '{}'"
-                logger.warning(_s.format(str(e)))
+                try:
+                    temporary_lm = LibraryManager(self.config.library_root)
+                    # try to initialize library
+                    temporary_lm.add_library(library)
+                    temporary_lm.update([library.name])
+                    # the initialization worked, now register it properly
+                    self.add_library(library)
+                except (RuntimeError, OSError) as e:
+                    _s = "Failed to register library '{}'"
+                    logger.warning(_s.format(str(e)))
 
     @staticmethod
     def init_logging(verbose, monochrome, log_file=None):
diff --git a/fusesoc/librarymanager.py b/fusesoc/librarymanager.py
index bbacceed..f1e638b6 100644
--- a/fusesoc/librarymanager.py
+++ b/fusesoc/librarymanager.py
@@ -50,16 +50,23 @@ def l(s):
             logger.info(l("sync-type is local. Ignoring update"))
             return
 
-        # FIXME: Do an initial checkout if missing
-        if not os.path.exists(self.location):
-            logger.warning(l(f"{self.location} does not exist. Ignoring update"))
-            return
-
         if not (self.auto_sync or force):
             logger.info(l("auto-sync disabled. Ignoring update"))
             return
 
         provider = get_provider(self.sync_type)
+
+        if not os.path.exists(self.location):
+            logger.info(l(f"{self.location} does not exist. Trying a checkout"))
+            try:
+                provider.init_library(self)
+            except RuntimeError as e:
+                # Keep old behavior of logging a warning if there is a library
+                # in `fusesoc.conf`, but the directory does not exist for some
+                # reason and it could not be initialized.
+                logger.warning(l(f"{self.location} does not exist. Ignoring update"))
+            return
+
         try:
             logger.info(l("Updating..."))
             provider.update_library(self)


Test patch:
diff --git a/tests/test_libraries.py b/tests/test_libraries.py
index b277e77b..90971f12 100644
--- a/tests/test_libraries.py
+++ b/tests/test_libraries.py
@@ -214,3 +214,36 @@ def test_library_update(caplog):
         fs.update_libraries([])
 
     assert "test_lib : sync-type is local. Ignoring update" in caplog.text
+
+
+def test_library_update_with_initialize(caplog):
+    with tempfile.TemporaryDirectory() as library:
+
+        with tempfile.NamedTemporaryFile(mode="w+") as tcf:
+            tcf.write(
+                f"""[main]
+library_root = {library}
+
+[library.vlog_tb_utils]
+location = fusesoc_libraries/vlog_tb_utils
+sync-uri = https://github.com/fusesoc/vlog_tb_utils
+sync-type = git
+auto-sync = true
+
+"""
+            )
+            tcf.flush()
+
+            conf = Config(tcf.name)
+
+        args = Namespace()
+
+        Fusesoc.init_logging(False, False)
+        fs = Fusesoc(conf)
+
+        with caplog.at_level(logging.INFO):
+            fs.update_libraries([])
+
+        assert "vlog_tb_utils does not exist. Trying a checkout" in caplog.text
+        assert "Cloning library into fusesoc_libraries/vlog_tb_utils" in caplog.text
+        assert "Updating..." in caplog.text