Repository: olofk/fusesoc

Issue:
How to use fusesoc.conf with RCS?
I've just started trying to use `fusesoc.conf` to manage libraries and am not sure I'm using it correctly.  My aim is to check in the configuration file such that when a user checks out a clean repo and uses `fusesoc`, all libraries that are missing get populated.

Running something like `fusesoc library add idex_generators https://github.com/idex-biometrics/fusesoc-generators` results in a configuration file as follows:

```
[library.idex_generators]
location = fusesoc_libraries/idex_generators
sync-uri = https://github.com/idex-biometrics/fusesoc-generators
sync-type = git
auto-sync = true
```

but if I commit that file and run `fusesoc` as normal I get a warning about:

```
WARNING: Failed to register library 'fusesoc_libraries/idex_generators is not a directory'
```

The `fusesoc library` sub-command does have an `update` argument but that seems to incorrectly identify the library as `local`:

```
$ fusesoc library update
WARNING: Failed to register library 'fusesoc_libraries/idex_generators is not a directory'
INFO: /home/shareefj/git/GN-651 : sync-type is local. Ignoring update
```

Perhaps that's the issue?  Is running `update` the correct way to populate the libraries but it's not working due to a bug in determining the `sync-type`?

Hints from issue comments:
@olofk: I guess the main problem is that checking in fusesoc.conf is not an intended use-case. It does seem like it _almost_ works though, so let's try to make what you're trying to do possible.

Yes, that looks like a bug. There are some heuristics involved here and I'm frankly not super familiar with that part of the code as I didn't write it myself. I think it's a fair assumption that `fusesoc library update`, but I wonder if that INFO message is really related to the same library as was mentioned in the warning.

What I guess happens is that FuseSoC tries to go into the library and run `git update` and fails because the library doesn't exist. I think we could change that to instead do a fresh clone in that case, which would probably solve your use-case@shareefj: So is the original use case that a user should always run `fusesoc library add` on each clean checkout?

I can take a closer look at the code and try and work out what's going on.@olofk: Library managment in FuseSoC could need some more love in general. The use-case you describe just wasn't considered by anyone. The assumption was that the workspace directory is a throw-away directory that isn't under version control. But I do see the benefit of checking in a base workspace dir, especially if you want to put some utilities and stuff inside that.

Solution patch:
diff --git a/fusesoc/capi2/json_schema.py b/fusesoc/capi2/json_schema.py
index 9066dd6..71e672f 100644
--- a/fusesoc/capi2/json_schema.py
+++ b/fusesoc/capi2/json_schema.py
@@ -1,27 +1,287 @@
-# Copyright FuseSoC contributors
-# Licensed under the 2-Clause BSD License, see LICENSE for details.
-# SPDX-License-Identifier: BSD-2-Clause
-
-# FIXME: Fix nicer method to read JSON schema
-
 capi2_schema = """
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "title": "CAPI2",
   "description": "Core API Version 2",
   "type": "object",
-  "patternProperties": {
-    "^name$": {
+  "properties": {
+    "description": {
+      "description": "Short description of core",
+      "type": "string"
+    },
+    "filesets": { "$ref": "#/$defs/filesets" },
+    "generate": { "$ref": "#/$defs/generate" },
+    "generators": { "$ref": "#/$defs/generators" },
+    "name": {
       "description": "VLNV identifier for core",
       "type": "string",
       "minProperties" : 1,
       "maxProperties" : 1
     },
-    "^description$": {
-      "description": "Short description of core",
-      "type": "string"
+    "parameters": { "$ref": "#/$defs/parameters" },
+    "provider": { "$ref": "#/$defs/provider" },
+    "scripts": { "$ref": "#/$defs/scripts" },
+    "targets": { "$ref": "#/$defs/targets" },
+    "vpi": {
+      "description": "A VPI (Verilog Procedural Interface) library is a shared object that is built and loaded by a simulator to provide extra Verilog system calls. This section describes what files and external libraries to use for building a VPI library",
+      "type": "object",
+      "patternProperties": {
+        "^.+$": {
+          "patternProperties": {
+            "^filesets(_append)?$": {
+              "description": "Filesets containing files to use when compiling the VPI library",
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            },
+            "^libs(_append)?$": {
+              "description": "External libraries to link against",
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            }
+          }
+        }
+      }
+    },
+    "virtual": {
+      "description": "VLNV of a virtual core provided by this core. Versions are currently not supported, only the VLN part is used.",
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    }
+  },
+  "additionalProperties": false,
+
+  "$defs": {
+
+    "files": {
+      "description": "Files in fileset",
+      "type": "array",
+      "minContains": 1,
+      "items": {
+        "oneOf": [
+          {
+            "type": "string"
+          },
+          {
+            "type": "object",
+            "minProperties": 1,
+            "maxProperties": 1,
+            "patternProperties": {
+              "^.+$": {
+                "description": "Path to file",
+                "type": "object",
+                "properties": {
+                  "define": {
+		    "description": "Defines to be used for this file. These defines will be added to those specified in the target parameters section. If a define is specified both here and in the target parameter section, the value specified here will take precedence.  The parameter default value can be set here with ``param=value``",
+		    "type": "object",
+		    "patternProperties": {
+		      "^.+$": {
+			"anyOf": [
+			  { "type": "string" },
+			  { "type": "number" },
+			  { "type": "boolean"}
+			]
+		      }
+		    }
+		  },
+                  "is_include_file": {
+                    "description": "Treats file as an include file when true",
+                    "type": "boolean"
+                  },
+                  "include_path": {
+                    "description": "Explicitly set an include directory, relative to core root, instead of the directory containing the file",
+                    "type": "string"
+                  },
+                  "file_type": {
+                    "description": "File type. Overrides the file_type set on the containing fileset",
+                    "type": "string"
+                  },
+                  "logical_name": {
+                    "description": "Logical name, i.e. library for VHDL/SystemVerilog. Overrides the logical_name set on the containing fileset",
+                    "type": "string"
+                  },
+                  "tags": {
+                    "description": "Tags, special file-specific hints for the backends. Appends the tags set on the containing fileset",
+                    "type": "array",
+                    "items": {
+                      "type": "string"
+                    }
+                  },
+                  "copyto": {
+                    "description": "Copy the source file to this path in the work directory",
+                    "type": "string"
+                  }
+                },
+                "additionalProperties": false
+              }
+            },
+            "additionalProperties": false
+          }
+        ]
+      }
+    },
+
+    "filesets": {
+      "description": "A fileset represents a group of files with a common purpose. Each file in the fileset is required to have a file type and is allowed to have a logical_name which can be set for the whole fileset or individually for each file. A fileset can also have dependencies on other cores, specified in the depend section",
+      "type": "object",
+      "patternProperties": {
+        "^.+$": {
+          "description": "Name of fileset",
+          "type": "object",
+          "patternProperties": {
+            "^file_type$": {
+              "description": "Default file_type for files in fileset",
+              "type": "string"
+            },
+            "^logical_name$": {
+              "description": "Default logical_name (i.e. library) for files in fileset",
+              "type": "string"
+            },
+            "^tags$": {
+              "description": "Default tags for files in fileset",
+               "type": "array",
+               "items": {
+                 "type": "string"
+               }
+            },
+            "^files(_append)?$": { "$ref": "#/$defs/files" },
+            "^depend(_append)?$": {
+              "description": "Dependencies of fileset",
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            }
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "generate": {
+      "description": "The elements in this section each describe a parameterized instance of a generator. They specify which generator to invoke and any generator-specific parameters",
+      "type": "object",
+      "patternProperties": {
+        "^.+$": {
+          "description": "Name of generator to use",
+          "properties": {
+            "generator": {
+              "description": "The generator to use. Note that the generator must be present in the dependencies of the core.",
+              "type": "string"
+            },
+            "position": {
+              "description": "Where to insert the generated core. Legal values are *first*, *prepend*, *append* or *last*. *prepend* (*append*) will insert core before (after) the core that called the generator",
+              "type": "string",
+              "pattern": "^first|prepend|append|last$"
+            },
+            "parameters": {
+              "description": "Generator-specific parameters. ``fusesoc gen show $generator`` might show available parameters. ",
+              "type": "object"
+            }
+          },
+          "additionalProperties": false,
+          "required": [
+            "generator"
+          ]
+        }
+      }
+    },
+
+    "generators": {
+      "description": "Generators are custom programs that generate FuseSoC cores. They are generally used during the build process, but can be used stand-alone too. This section allows a core to register a generator that can be used by other cores.",
+      "type": "object",
+      "patternProperties": {
+        "^.+$": {
+          "description": "Name of generator",
+          "properties": {
+            "command": {
+              "description": "The command to run (relative to the core root)",
+              "type": "string"
+            },
+            "interpreter": {
+              "description": "If the command needs a custom interpreter (such as python) this will be inserted as the first argument before command when calling the generator. The interpreter needs to be on the system PATH; specifically, shutil.which needs to be able to find the interpreter).",
+              "type": "string"
+            },
+            "cache_type": {
+              "description": "If the result of the generator should be considered cacheable. Legal values are *none*, *input* or *generator*.",
+              "type": "string",
+              "pattern": "^none|input|generator$"
+            },
+            "file_input_parameters": {
+              "description": "All parameters that are file inputs to the generator. This option can be used when *cache_type* is set to *input* if fusesoc should track if these files change.",
+              "type": "string"
+            },
+            "description": {
+              "description": "Short description of the generator, as shown with ``fusesoc gen list``",
+              "type": "string"
+            },
+            "usage": {
+              "description": "A longer description of how to use the generator, including which parameters it uses (as shown with ``fusesoc gen show $generator``)",
+              "type": "string"
+            }
+          },
+          "additionalProperties": false,
+          "required": [
+            "command"
+          ]
+        }
+      }
     },
-    "^provider$": {
+
+    "parameters": {
+      "description": "Available parameters",
+      "type": "object",
+      "patternProperties": {
+        "^.+$": {
+          "properties": {
+            "datatype": {
+              "description": "Parameter datatype. Legal values are *bool*, *file*, *int*, *str*. *file* is same as *str*, but prefixed with the current directory that FuseSoC runs from",
+              "type": "string",
+              "pattern": "^bool|file|int|real|str$"
+            },
+            "default": {
+              "description": "Default value",
+              "oneOf": [
+                {
+                  "type": "boolean"
+                },
+                {
+                  "type": "string"
+                },
+                {
+                  "type": "number"
+                }
+              ]
+            },
+            "description": {
+              "description": "Description of the parameter, as can be seen with ``fusesoc run --target=$target $core --help``",
+              "type": "string"
+            },
+            "paramtype": {
+              "description": "Specifies type of parameter. Legal values are *cmdlinearg* for command-line arguments directly added when running the core, *generic* for VHDL generics, *plusarg* for verilog plusargs, *vlogdefine* for Verilog `` `define`` or *vlogparam* for verilog top-level parameters. All paramtypes are not valid for every backend. Consult the backend documentation for details.",
+              "type": "string"
+            },
+            "scope": {
+              "description": "**Not used** : Kept for backwards compatibility",
+              "type": "string"
+            }
+          },
+          "additionalProperties": false,
+          "required": [
+            "datatype",
+            "paramtype"
+          ]
+        }
+      }
+    },
+
+    "provider": {
       "description": "Provider of core",
       "type": "object",
       "anyOf": [
@@ -220,184 +480,7 @@ capi2_schema = """
         }
       ]
     },
-    "^filesets$": {
-      "description": "A fileset represents a group of files with a common purpose. Each file in the fileset is required to have a file type and is allowed to have a logical_name which can be set for the whole fileset or individually for each file. A fileset can also have dependencies on other cores, specified in the depend section",
-      "type": "object",
-      "patternProperties": {
-        "^.+$": {
-          "description": "Name of fileset",
-          "type": "object",
-          "patternProperties": {
-            "^file_type$": {
-              "description": "Default file_type for files in fileset",
-              "type": "string"
-            },
-            "^logical_name$": {
-              "description": "Default logical_name (i.e. library) for files in fileset",
-              "type": "string"
-            },
-            "^tags$": {
-              "description": "Default tags for files in fileset",
-               "type": "array",
-               "items": {
-                 "type": "string"
-               }
-            },
-            "^files(_append)?$": {
-              "description": "Files in fileset",
-              "type": "array",
-              "minContains": 1,
-              "items": {
-                "oneOf": [
-                  {
-                    "type": "string"
-                  },
-                  {
-                    "type": "object",
-                    "minProperties": 1,
-                    "maxProperties": 1,
-                    "patternProperties": {
-                      "^.+$": {
-                        "description": "Path to file",
-                        "type": "object",
-                        "properties": {
-                          "define": {
-                            "description": "Defines to be used for this file. These defines will be added to those specified in the target parameters section. If a define is specified both here and in the target parameter section, the value specified here will take precedence.  The parameter default value can be set here with ``param=value``",
-                            "type": "object",
-                            "patternProperties": {
-                              "^.+$": {
-                                "anyOf": [
-                                  {
-                                    "type": "string"
-                                  },
-                                  {
-                                    "type": "number"
-                                  },
-                                  {
-                                    "type": "boolean"
-                                  },
-                                ]
-                              }
-                            }
-                          },
-                          "is_include_file": {
-                            "description": "Treats file as an include file when true",
-                            "type": "boolean"
-                          },
-                          "include_path": {
-                            "description": "Explicitly set an include directory, relative to core root, instead of the directory containing the file",
-                            "type": "string"
-                          },
-                          "file_type": {
-                            "description": "File type. Overrides the file_type set on the containing fileset",
-                            "type": "string"
-                          },
-                          "logical_name": {
-                            "description": "Logical name, i.e. library for VHDL/SystemVerilog. Overrides the logical_name set on the containing fileset",
-                            "type": "string"
-                          },
-                          "tags": {
-                            "description": "Tags, special file-specific hints for the backends. Appends the tags set on the containing fileset",
-                            "type": "array",
-                            "items": {
-                              "type": "string"
-                            }
-                          },
-                          "copyto": {
-                            "description": "Copy the source file to this path in the work directory",
-                            "type": "string"
-                          }
-                        },
-                        "additionalProperties": false
-                      }
-                    },
-                    "additionalProperties": false
-                  }
-                ]
-              }
-            },
-            "^depend(_append)?$": {
-              "description": "Dependencies of fileset",
-              "type": "array",
-              "items": {
-                "type": "string"
-              }
-            }
-          },
-          "additionalProperties": false
-        }
-      },
-      "additionalProperties": false
-    },
-    "^generate$": {
-      "description": "The elements in this section each describe a parameterized instance of a generator. They specify which generator to invoke and any generator-specific parameters",
-      "type": "object",
-      "patternProperties": {
-        "^.+$": {
-          "description": "Name of generator to use",
-          "properties": {
-            "generator": {
-              "description": "The generator to use. Note that the generator must be present in the dependencies of the core.",
-              "type": "string"
-            },
-            "position": {
-              "description": "Where to insert the generated core. Legal values are *first*, *prepend*, *append* or *last*. *prepend* (*append*) will insert core before (after) the core that called the generator",
-              "type": "string",
-              "pattern": "^first|prepend|append|last$"
-            },
-            "parameters": {
-              "description": "Generator-specific parameters. ``fusesoc gen show $generator`` might show available parameters. ",
-              "type": "object"
-            }
-          },
-          "additionalProperties": false,
-          "required": [
-            "generator"
-          ]
-        }
-      }
-    },
-    "^generators$": {
-      "description": "Generators are custom programs that generate FuseSoC cores. They are generally used during the build process, but can be used stand-alone too. This section allows a core to register a generator that can be used by other cores.",
-      "type": "object",
-      "patternProperties": {
-        "^.+$": {
-          "description": "Name of generator",
-          "properties": {
-            "command": {
-              "description": "The command to run (relative to the core root)",
-              "type": "string"
-            },
-            "interpreter": {
-              "description": "If the command needs a custom interpreter (such as python) this will be inserted as the first argument before command when calling the generator. The interpreter needs to be on the system PATH; specifically, shutil.which needs to be able to find the interpreter).",
-              "type": "string"
-            },
-            "cache_type": {
-              "description": "If the result of the generator should be considered cacheable. Legal values are *none*, *input* or *generator*.",
-              "type": "string",
-              "pattern": "^none|input|generator$"
-            },
-            "file_input_parameters": {
-              "description": "All parameters that are file inputs to the generator. This option can be used when *cache_type* is set to *input* if fusesoc should track if these files change.",
-              "type": "string"
-            },
-            "description": {
-              "description": "Short description of the generator, as shown with ``fusesoc gen list``",
-              "type": "string"
-            },
-            "usage": {
-              "description": "A longer description of how to use the generator, including which parameters it uses (as shown with ``fusesoc gen show $generator``)",
-              "type": "string"
-            }
-          },
-          "additionalProperties": false,
-          "required": [
-            "command"
-          ]
-        }
-      }
-    },
-    "^scripts$": {
+    "scripts": {
       "description": "A script specifies how to run an external command that is called by the hooks section together with the actual files needed to run the script. Scripts are always executed from the work root",
       "type": "object",
       "patternProperties": {
@@ -431,7 +514,8 @@ capi2_schema = """
         }
       }
     },
-    "^targets$": {
+
+    "targets": {
       "description": "A target is the entry point to a core. It describes a single use-case and what resources that are needed from the core such as file sets, generators, parameters and specific tool options. A core can have multiple targets, e.g. for simulation, synthesis or when used as a dependency for another core. When a core is used, only a single target is active. The *default* target is a special target that is always used when the core is being used as a dependency for another core or when no ``--target=`` flag is set.",
       "type": "object",
       "patternProperties": {
@@ -625,84 +709,6 @@ capi2_schema = """
         }
       }
     },
-    "^parameters$": {
-      "description": "Available parameters",
-      "type": "object",
-      "patternProperties": {
-        "^.+$": {
-          "properties": {
-            "datatype": {
-              "description": "Parameter datatype. Legal values are *bool*, *file*, *int*, *str*. *file* is same as *str*, but prefixed with the current directory that FuseSoC runs from",
-              "type": "string",
-              "pattern": "^bool|file|int|real|str$"
-            },
-            "default": {
-              "description": "Default value",
-              "oneOf": [
-                {
-                  "type": "boolean"
-                },
-                {
-                  "type": "string"
-                },
-                {
-                  "type": "number"
-                }
-              ]
-            },
-            "description": {
-              "description": "Description of the parameter, as can be seen with ``fusesoc run --target=$target $core --help``",
-              "type": "string"
-            },
-            "paramtype": {
-              "description": "Specifies type of parameter. Legal values are *cmdlinearg* for command-line arguments directly added when running the core, *generic* for VHDL generics, *plusarg* for verilog plusargs, *vlogdefine* for Verilog `` `define`` or *vlogparam* for verilog top-level parameters. All paramtypes are not valid for every backend. Consult the backend documentation for details.",
-              "type": "string"
-            },
-            "scope": {
-              "description": "**Not used** : Kept for backwards compatibility",
-              "type": "string"
-            }
-          },
-          "additionalProperties": false,
-          "required": [
-            "datatype",
-            "paramtype"
-          ]
-        }
-      }
-    },
-    "^vpi$": {
-      "description": "A VPI (Verilog Procedural Interface) library is a shared object that is built and loaded by a simulator to provide extra Verilog system calls. This section describes what files and external libraries to use for building a VPI library",
-      "type": "object",
-      "patternProperties": {
-        "^.+$": {
-          "patternProperties": {
-            "^filesets(_append)?$": {
-              "description": "Filesets containing files to use when compiling the VPI library",
-              "type": "array",
-              "items": {
-                "type": "string"
-              }
-            },
-            "^libs(_append)?$": {
-              "description": "External libraries to link against",
-              "type": "array",
-              "items": {
-                "type": "string"
-              }
-            }
-          }
-        }
-      }
-    },
-    "^virtual(_append)?$": {
-      "description": "VLNV of a virtual core provided by this core. Versions are currently not supported, only the VLN part is used.",
-      "type": "array",
-      "items": {
-        "type": "string"
-      }
-    }
-  },
-  "additionalProperties": false
+  }
 }
 """
diff --git a/fusesoc/edalizer.py b/fusesoc/edalizer.py
index 7dce442..30918c4 100644
--- a/fusesoc/edalizer.py
+++ b/fusesoc/edalizer.py
@@ -544,9 +544,10 @@ class Ttptttg:
         }
 
     def _sha256_input_yaml_hexdigest(self):
-        return hashlib.sha256(
-            utils.yaml_dump(self.generator_input).encode()
-        ).hexdigest()
+        data = self.generator_input.copy()
+        # Remove files_root since that is not deterministic
+        data.pop("files_root")
+        return hashlib.sha256(utils.yaml_dump(data).encode()).hexdigest()
 
     def _sha256_file_input_hexdigest(self):
         input_files = []
@@ -580,17 +581,6 @@ class Ttptttg:
 
         return hash.hexdigest()
 
-    def _fwrite_hash(self, hashfile, data):
-        with open(hashfile, "w") as f:
-            f.write(data)
-
-    def _fread_hash(self, hashfile):
-        data = ""
-        with open(hashfile) as f:
-            data = f.read()
-
-        return data
-
     def _run(self, generator_cwd):
         logger.info("Generating " + str(self.vlnv))
 
@@ -632,6 +622,11 @@ class Ttptttg:
     def is_cacheable(self):
         return self.is_input_cacheable() or self.is_generator_cacheable()
 
+    def acquire_cache_lock(self):
+        have_lock = False
+        # while not have_lock:
+        #    if
+
     def generate(self):
         """Run a parametrized generator
 
@@ -641,7 +636,7 @@ class Ttptttg:
 
         hexdigest = self._sha256_input_yaml_hexdigest()
 
-        logger.debug("Generator input yaml hash: " + hexdigest)
+        logger.debug("Generator parameters hash: " + hexdigest)
 
         generator_cwd = os.path.join(
             self.gen_root,
@@ -649,6 +644,16 @@ class Ttptttg:
             self.vlnv.sanitized_name + "-" + hexdigest,
         )
 
+        if "file_input_parameters" in self.generator:
+            # If file_input_parameters has been configured in the generator
+            # parameters will be iterated to look for files to add to the
+            # input files hash calculation.
+            file_input_hash = self._sha256_file_input_hexdigest()
+            generator_cwd = os.path.join(generator_cwd, file_input_hash)
+            logger.debug("Generator input files hash: " + file_input_hash)
+
+        logger.debug("Generator cwd: " + generator_cwd)
+
         if os.path.lexists(generator_cwd) and not os.path.isdir(generator_cwd):
             raise RuntimeError(
                 "Unable to create generator working directory since it already exists and is not a directory: "
@@ -657,59 +662,18 @@ class Ttptttg:
                 + "Remove it manually or run 'fusesoc gen clean'"
             )
 
-        if self.is_input_cacheable():
-            # Input cache enabled. Check if cached output already exists in generator_cwd.
-            logger.debug("Input cache enabled.")
-
-            # If file_input_parameters has been configured in the generator
-            # parameters will be iterated to look for files to add to the
-            # input files hash calculation.
-            if "file_input_parameters" in self.generator:
-                file_input_hash = self._sha256_file_input_hexdigest()
-
-                logger.debug("Generator file input hash: " + file_input_hash)
-
-                hashfile = os.path.join(generator_cwd, ".fusesoc_file_input_hash")
-
-                rerun = False
-
-                if os.path.isfile(hashfile):
-                    cached_hash = self._fread_hash(hashfile)
-                    logger.debug("Cached file input hash: " + cached_hash)
-
-                    if not file_input_hash == cached_hash:
-                        logger.debug("File input has changed.")
-                        rerun = True
-                    else:
-                        logger.info("Found cached output for " + str(self.vlnv))
-
-                else:
-                    logger.debug("File input hash file does not exist: " + hashfile)
-                    rerun = True
-
-                if rerun:
-                    shutil.rmtree(generator_cwd, ignore_errors=True)
-                    self._run(generator_cwd)
-                    self._fwrite_hash(hashfile, file_input_hash)
-
-            elif os.path.isdir(generator_cwd):
-                logger.info("Found cached output for " + str(self.vlnv))
-            else:
-                # No directory found. Run generator.
-                self._run(generator_cwd)
-
-        elif self.is_generator_cacheable():
-            # Generator cache enabled. Call the generator and let it
-            # decide if the old output still is valid.
-            logger.debug("Generator cache enabled.")
-            self._run(generator_cwd)
+        if self.is_input_cacheable() and os.path.isdir(generator_cwd):
+            logger.info("Found cached output for " + str(self.vlnv))
         else:
-            # No caching enabled. Try to remove directory if it already exists.
-            # This could happen if a generator that has been configured with
-            # caching is changed to no caching.
-            logger.debug("Generator cache is not enabled.")
-            shutil.rmtree(generator_cwd, ignore_errors=True)
+            # TODO: Acquire a lock here to ensure that we are the only users
+            if self.is_generator_cacheable():
+                logger.warning(
+                    "Support for generator-side cachable cores are deprecated and will be removed"
+                )
+            else:
+                shutil.rmtree(generator_cwd, ignore_errors=True)
             self._run(generator_cwd)
+            # TODO: Release cache lock
 
         cores = []
         logger.debug("Looking for generated or cached cores in " + generator_cwd)
diff --git a/fusesoc/fusesoc.py b/fusesoc/fusesoc.py
index 37cc25e..5f163fc 100644
--- a/fusesoc/fusesoc.py
+++ b/fusesoc/fusesoc.py
@@ -37,8 +37,16 @@ class Fusesoc:
             try:
                 self.add_library(library)
             except (RuntimeError, OSError) as e:
-                _s = "Failed to register library '{}'"
-                logger.warning(_s.format(str(e)))
+                try:
+                    temporary_lm = LibraryManager(self.config.library_root)
+                    # try to initialize library
+                    temporary_lm.add_library(library)
+                    temporary_lm.update([library.name])
+                    # the initialization worked, now register it properly
+                    self.add_library(library)
+                except (RuntimeError, OSError) as e:
+                    _s = "Failed to register library '{}'"
+                    logger.warning(_s.format(str(e)))
 
     @staticmethod
     def init_logging(verbose, monochrome, log_file=None):
diff --git a/fusesoc/librarymanager.py b/fusesoc/librarymanager.py
index bbaccee..f1e638b 100644
--- a/fusesoc/librarymanager.py
+++ b/fusesoc/librarymanager.py
@@ -50,16 +50,23 @@ class Library:
             logger.info(l("sync-type is local. Ignoring update"))
             return
 
-        # FIXME: Do an initial checkout if missing
-        if not os.path.exists(self.location):
-            logger.warning(l(f"{self.location} does not exist. Ignoring update"))
-            return
-
         if not (self.auto_sync or force):
             logger.info(l("auto-sync disabled. Ignoring update"))
             return
 
         provider = get_provider(self.sync_type)
+
+        if not os.path.exists(self.location):
+            logger.info(l(f"{self.location} does not exist. Trying a checkout"))
+            try:
+                provider.init_library(self)
+            except RuntimeError as e:
+                # Keep old behavior of logging a warning if there is a library
+                # in `fusesoc.conf`, but the directory does not exist for some
+                # reason and it could not be initialized.
+                logger.warning(l(f"{self.location} does not exist. Ignoring update"))
+            return
+
         try:
             logger.info(l("Updating..."))
             provider.update_library(self)


Test patch:
diff --git a/tests/test_capi2.py b/tests/test_capi2.py
index 89ae458..687b994 100644
--- a/tests/test_capi2.py
+++ b/tests/test_capi2.py
@@ -324,11 +324,13 @@ def test_capi2_get_generators():
     core = Core(Core2Parser(), os.path.join(cores_dir, "generate", "generators.core"))
 
     generators = core.get_generators()
-    assert len(generators) == 2
+    assert len(generators) == 3
     assert generators["generator1"]["command"] == "testgen.py"
     assert generators["generator2"]["command"] == "testgen.py"
     assert generators["generator2"]["cache_type"] == "input"
-    assert generators["generator2"]["file_input_parameters"] == "file_in_param1"
+    assert generators["generator3"]["command"] == "testgen.py"
+    assert generators["generator3"]["cache_type"] == "input"
+    assert generators["generator3"]["file_input_parameters"] == "file_in_param1"
 
 
 def test_capi2_get_parameters():
@@ -606,6 +608,12 @@ def test_capi2_get_ttptttg():
             "name": "testgenerate_with_cache",
             "generator": "generator2",
             "pos": "append",
+            "config": {"some_option": "some_value"},
+        },
+        {
+            "name": "testgenerate_with_file_cache",
+            "generator": "generator3",
+            "pos": "append",
             "config": {"file_in_param1": "file_cachetest"},
         },
     ]
diff --git a/tests/test_edalizer.py b/tests/test_edalizer.py
index 6797730..03ef20e 100644
--- a/tests/test_edalizer.py
+++ b/tests/test_edalizer.py
@@ -148,9 +148,9 @@ def test_tool_or_flow():
 
 
 def test_generators():
-    import os
     import shutil
     import tempfile
+    from pathlib import Path
 
     from fusesoc.config import Config
     from fusesoc.coremanager import CoreManager
@@ -158,8 +158,8 @@ def test_generators():
     from fusesoc.librarymanager import Library
     from fusesoc.vlnv import Vlnv
 
-    tests_dir = os.path.dirname(__file__)
-    cores_dir = os.path.join(tests_dir, "capi2_cores", "misc", "generate")
+    tests_dir = Path(__file__).parent
+    cores_dir = tests_dir / "capi2_cores" / "misc" / "generate"
 
     lib = Library("edalizer", cores_dir)
 
@@ -168,14 +168,14 @@ def test_generators():
 
     core = cm.get_core(Vlnv("::generate"))
 
-    build_root = tempfile.mkdtemp(prefix="export_")
-    export_root = os.path.join(build_root, "exported_files")
+    build_root = Path(tempfile.mkdtemp(prefix="export_"))
+    export_root = build_root / "exported_files"
 
     edalizer = Edalizer(
         toplevel=core.name,
         flags={"tool": "icarus"},
         core_manager=cm,
-        work_root=os.path.join(build_root, "work"),
+        work_root=build_root / "work",
         export_root=export_root,
         system_name=None,
     )
@@ -193,11 +193,13 @@ def test_generators():
                 "::generate-testgenerate_with_params:0",
                 "::generate-testgenerate_with_override:0",
                 "::generate-testgenerate_with_cache:0",
+                "::generate-testgenerate_with_file_cache:0",
             ],
             "::generate-testgenerate_without_params:0": [],
             "::generate-testgenerate_with_params:0": [],
             "::generate-testgenerate_with_override:0": [],
             "::generate-testgenerate_with_cache:0": [],
+            "::generate-testgenerate_with_file_cache:0": [],
         },
         "parameters": {"p": {"datatype": "str", "paramtype": "vlogparam"}},
         "tool_options": {"icarus": {}},
@@ -209,7 +211,6 @@ def test_generators():
     }
 
     assert ref_edam == edalizer.edam
-    edalizer.export()
 
     name_to_core = {str(core.name): core for core in edalizer.cores}
     for flavour in ["testgenerate_with_params", "testgenerate_without_params"]:
@@ -217,17 +218,32 @@ def test_generators():
         assert core_name in name_to_core
         core = name_to_core[core_name]
 
-    # Test generator input cache and file_input_params
-    core_name = f"::generate-testgenerate_with_cache:0"
+    # Test generator input without file_input_params
+    core_name = "::generate-testgenerate_with_cache:0"
     assert core_name in name_to_core
     core = name_to_core[core_name]
-    assert os.path.isdir(core.core_root)
 
-    hash = ""
-    with open(os.path.join(core.core_root, ".fusesoc_file_input_hash")) as f:
-        hash = f.read()
+    core_root = Path(core.core_root)
+    assert core_root.is_dir()
+    assert (
+        core_root.name
+        == "generate-testgenerate_with_cache_0-616d6cf151dba72fcd893c08a8e18e6dba2b81ee25dec08c92e0177064dfc18c"
+    )
+    shutil.rmtree(core.core_root, ignore_errors=True)
 
-    # SHA256 hash of test file content
-    assert hash == "da265f9dccc9d9e64d059f677508f9550b403c99e6ce5df07c6fb1d711d0ee99"
+    # Test generator input file_input_params
+    core_name = "::generate-testgenerate_with_file_cache:0"
+    assert core_name in name_to_core
+    core = name_to_core[core_name]
+    core_root = Path(core.core_root)
 
+    assert core_root.is_dir()
+    assert (
+        core_root.name
+        == "da265f9dccc9d9e64d059f677508f9550b403c99e6ce5df07c6fb1d711d0ee99"
+    )
+    assert (
+        core_root.parent.name
+        == "generate-testgenerate_with_file_cache_0-f3d9e1e462ef1f7113fafbacd62d6335dd684e69332f75498fb01bfaaa7c11ee"
+    )
     shutil.rmtree(core.core_root, ignore_errors=True)
diff --git a/tests/test_libraries.py b/tests/test_libraries.py
index b277e77..90971f1 100644
--- a/tests/test_libraries.py
+++ b/tests/test_libraries.py
@@ -214,3 +214,36 @@ def test_library_update(caplog):
         fs.update_libraries([])
 
     assert "test_lib : sync-type is local. Ignoring update" in caplog.text
+
+
+def test_library_update_with_initialize(caplog):
+    with tempfile.TemporaryDirectory() as library:
+
+        with tempfile.NamedTemporaryFile(mode="w+") as tcf:
+            tcf.write(
+                f"""[main]
+library_root = {library}
+
+[library.vlog_tb_utils]
+location = fusesoc_libraries/vlog_tb_utils
+sync-uri = https://github.com/fusesoc/vlog_tb_utils
+sync-type = git
+auto-sync = true
+
+"""
+            )
+            tcf.flush()
+
+            conf = Config(tcf.name)
+
+        args = Namespace()
+
+        Fusesoc.init_logging(False, False)
+        fs = Fusesoc(conf)
+
+        with caplog.at_level(logging.INFO):
+            fs.update_libraries([])
+
+        assert "vlog_tb_utils does not exist. Trying a checkout" in caplog.text
+        assert "Cloning library into fusesoc_libraries/vlog_tb_utils" in caplog.text
+        assert "Updating..." in caplog.text