Repository: gleam-lang/gleam (rust)

Issue:
Caching problem when updating dependencies
I found a reproduction for a build directory caching problem using [this repo](https://github.com/fawni/june).

Inside of the repository, running these commands:

```bash
rm -fr build
# last "good" commit
git checkout e2ceac56ba5afa9b7a27d8a08e70e2cee619d15d; gleam build
# update all dependencies
git checkout 18bda7afecb40f5425ed47c79722cd7ccbdefeba; gleam build
# fix some of the errors (nakai), but not all of them (mist/wisp)
git checkout ecc3a5f92a46312b429e3736dd8cb370f5c744ae; gleam build
```

produces:

```
removed directory 'build'
Previous HEAD position was ecc3a5f 🐛 fix: comply with latest nakai api
HEAD is now at e2ceac5 ✨ feature: delete files
   [...gleam build output stripped...]
   Compiled in 120.14s
Previous HEAD position was e2ceac5 ✨ feature: delete files
HEAD is now at 18bda7a 📦 deps: bump everything
Downloading packages
 Downloaded 18 packages in 0.21s
 [...gleam build output stripped...]
error: Incorrect arity
   ┌─ /home/arkan/Projects/probe/june/src/june/pages.gleam:29:9
   │
29 │         attr.crossorigin(),
   │         ^^^^^^^^^^^^^^^^^^ Expected 1 argument, got 0


Previous HEAD position was 18bda7a 📦 deps: bump everything
HEAD is now at ecc3a5f 🐛 fix: comply with latest nakai api
  Compiling june
error: Type mismatch
   ┌─ /home/arkan/Projects/probe/june/src/june/router.gleam:24:29
   │
24 │   use req <- web.middleware(req)
   │                             ^^^

Expected type:

    Request(Connection)

Found type:

    Request(wisp/internal.Connection)

error: Type mismatch
   ┌─ /home/arkan/Projects/probe/june/src/june/router.gleam:27:23
   │
27 │     [] -> handle_root(req)
   │                       ^^^

Expected type:

    Request(Connection)

Found type:

    Request(wisp.Connection)

error: Type mismatch
   ┌─ /home/arkan/Projects/probe/june/src/june/router.gleam:28:31
   │
28 │     _ -> handle_retrieve_file(req)
   │                               ^^^

Expected type:

    Request(Connection)

Found type:

    Request(wisp.Connection)
```

removing the build directory again and rebuilding reveals another type error and gets rid of the errors above:

```
error: Unknown module value
   ┌─ /home/arkan/Projects/probe/june/src/june.gleam:17:9
   │
17 │     wisp.mist_handler(router.handle_request, token)
   │         ^^^^^^^^^^^^^

The module `wisp` does not have a `mist_handler` value.
```

Just guessing here, but I think it has to do with the fact that type checking stops after finding that first nakai-related error, but the compiler then assumes this was the only dependency-related problem.

Gleam 1.5.1 ~ :purple_heart: 

Pull request:
Fix #3772: Delete cached metadata if a module is stale because of it's dependencies
This fixes both caching problems described in #3772 by deleting the cache metadata if a module is found to be stale because of its dependencies.

What happened was that the dependency introduced breaking changes but otherwise compiled fine; this meant that the dependent module produced build errors but still had valid cache files. On the next run, the build cache was valid and the dependency no longer stale, so the build tool assumed everything was fine.

I copied the way the path to the `.cache_meta` file is constructed from `ModuleLoader`. I did not like putting it into `module_loader.rs` personally, but I wouldn't mind either way, please tell me if I should move stuff/make it more dry :)

~ :purple_heart: 


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5de6813ff..75fff529f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,9 @@
   emitted when the enclosing function has an external JavaScript implementation.
   ([Richard Viney](https://github.com/richard-viney))
 
+- Fixed a bug where updating a dependency could break the build cache.
+  ([yoshi](https://github.com/joshi-monster))
+
 ## v1.6.0-rc1 - 2024-11-10
 
 ### Build tool
diff --git a/README.md b/README.md
index 6b38e184b..3862240d7 100644
--- a/README.md
+++ b/README.md
@@ -62,6 +62,7 @@ Thank you to our sponsors! Gleam would not be possible without you.
   <a href="https://github.com/benmyles">Ben Myles</a> -
   <a href="https://github.com/bcpeinhardt">Benjamin Peinhardt</a> -
   <a href="https://github.com/bentomas">Benjamin Thomas</a> -
+  <a href="https://github.com/berkanteber">Berkan Teber</a> -
   <a href="https://github.com/bgwdotdev">bgw</a> -
   <a href="https://github.com/bigtallbill">Bill Nunney</a> -
   <a href="https://github.com/bjartelund">Bjarte Aarmo Lund</a> -
@@ -107,7 +108,6 @@ Thank you to our sponsors! Gleam would not be possible without you.
   <a href="https://github.com/DanielleMaywood">Danielle Maywood</a> -
   <a href="https://github.com/pinnet">Danny Arnold</a> -
   <a href="https://github.com/despairblue">Danny Martini</a> -
-  <a href="https://github.com/dar5hak">Darshak Parikh</a> -
   <a href="https://github.com/davydog187">Dave Lucia</a> -
   <a href="https://github.com/dbernheisel">David Bernheisel</a> -
   <a href="https://github.com/davidcornu">David Cornu</a> -
@@ -222,6 +222,7 @@ Thank you to our sponsors! Gleam would not be possible without you.
   <a href="https://github.com/krystofrezac">Kryštof Řezáč</a> -
   <a href="https://github.com/krzysztofgb">Krzysztof G.</a> -
   <a href="https://github.com/leostera">Leandro Ostera</a> -
+  <a href="https://github.com/leejarvis">Lee Jarvis</a> -
   <a href="https://github.com/leonqadirie">Leon Qadirie</a> -
   <a href="https://github.com/LeartS">Leonardo Donelli</a> -
   <a href="https://github.com/defp">lidashuang</a> -
@@ -287,7 +288,6 @@ Thank you to our sponsors! Gleam would not be possible without you.
   <a href="https://github.com/Davorak">Patrick Wheeler</a> -
   <a href="https://github.com/giddie">Paul Gideon Dann</a> -
   <a href="https://github.com/pguse">Paul Guse</a> -
-  <a href="https://github.com/vidalpaul">Paulo Vidal</a> -
   <a href="https://github.com/biernacki">Pawel Biernacki</a> -
   <a href="https://github.com/Tulkdan">Pedro Correa</a> -
   <a href="https://github.com/petejodo">Pete Jodo</a> -
@@ -299,6 +299,7 @@ Thank you to our sponsors! Gleam would not be possible without you.
   <a href="https://github.com/qdentity">Qdentity</a> -
   <a href="https://github.com/qingliangcn">qingliangcn</a> -
   <a href="https://github.com/raquentin">Race Williams</a> -
+  <a href="https://github.com/stoft">Rasmus</a> -
   <a href="https://github.com/ray-delossantos">Ray</a> -
   <a href="https://github.com/chouzar">Raúl Chouza </a> -
   <a href="https://github.com/renatillas">re.natillas</a> -
@@ -313,9 +314,9 @@ Thank you to our sponsors! Gleam would not be possible without you.
   <a href="https://github.com/TanklesXL">Robert Attard</a> -
   <a href="https://github.com/rellen">Robert Ellen</a> -
   <a href="https://github.com/malkomalko">Robert Malko</a> -
-  <a href="https://github.com/R0DR160HM">Rodrigo Heinzen de Moraes</a> -
   <a href="https://github.com/Papipo">Rodrigo Álvarez</a> -
   <a href="https://liberapay.com/Karakunai/">Ronan Harris</a> -
+  <a href="https://github.com/rotabull">Rotabull</a> -
   <a href="https://github.com/reinefjord">Rupus Reinefjord</a> -
   <a href="https://github.com/ustitc">Ruslan Ustitc</a> -
   <a href="https://github.com/samaaron">Sam Aaron</a> -
@@ -371,5 +372,6 @@ Thank you to our sponsors! Gleam would not be possible without you.
   <a href="https://github.com/Yasuo-Higano">Yasuo Higano</a> -
   <a href="https://github.com/joshi-monster">yoshi~ </a> -
   <a href="https://github.com/gasparinzsombor">Zsombor Gasparin</a> -
-  <a href="https://liberapay.com/~1847917/">~1847917</a>
+  <a href="https://liberapay.com/~1847917/">~1847917</a> -
+  <a href="https://github.com/eberfreitas">Éber Freitas Dias</a>
 </p>
\ No newline at end of file
diff --git a/compiler-core/src/build/package_loader.rs b/compiler-core/src/build/package_loader.rs
index 77259cac2..ad7335a93 100644
--- a/compiler-core/src/build/package_loader.rs
+++ b/compiler-core/src/build/package_loader.rs
@@ -140,7 +140,7 @@ where
             match input {
                 // A new uncached module is to be compiled
                 Input::New(module) => {
-                    tracing::debug!(module = %module.name, "module_to_be_compiled");
+                    tracing::debug!(module = %module.name, "new_module_to_be_compiled");
                     self.stale_modules.add(module.name.clone());
                     loaded.to_compile.push(module);
                 }
@@ -149,9 +149,9 @@ where
                 // recompiled as the changes in the dependencies may have affect
                 // the output, making the cache invalid.
                 Input::Cached(info) if self.stale_modules.includes_any(&info.dependencies) => {
-                    tracing::debug!(module = %info.name, "module_to_be_compiled");
+                    tracing::debug!(module = %info.name, "stale_module_to_be_compiled");
                     self.stale_modules.add(info.name.clone());
-                    let module = self.load_and_parse(info)?;
+                    let module = self.load_stale_module(info)?;
                     loaded.to_compile.push(module);
                 }
 
@@ -280,8 +280,22 @@ where
         Ok(inputs.collection)
     }
 
-    fn load_and_parse(&self, cached: CachedModule) -> Result<UncompiledModule> {
+    fn load_stale_module(&self, cached: CachedModule) -> Result<UncompiledModule> {
         let mtime = self.io.modification_time(&cached.source_path)?;
+
+        // We need to delete any existing cache_meta files for this module.
+        // While we figured it out this time because the module has stale dependencies,
+        // next time the dependencies might no longer be stale, but we still need to be able to tell
+        // that this module needs to be recompiled until it sucessfully compiles at least once.
+        // This can happen if the stale dependency includes breaking changes.
+        let artefact = cached.name.replace("/", "@");
+        let meta_path = self
+            .artefact_directory
+            .join(artefact.as_str())
+            .with_extension("cache_meta");
+
+        let _ = self.io.delete_file(&meta_path);
+
         read_source(
             self.io.clone(),
             self.target,


Test patch:
diff --git a/compiler-core/src/build/package_loader/tests.rs b/compiler-core/src/build/package_loader/tests.rs
index a29ec58ba..6eece9e24 100644
--- a/compiler-core/src/build/package_loader/tests.rs
+++ b/compiler-core/src/build/package_loader/tests.rs
@@ -218,6 +218,43 @@ fn module_is_stale_if_deps_are_stale() {
     assert_eq!(loaded.cached, vec![EcoString::from("three")]);
 }
 
+#[test]
+fn module_continues_to_be_stale_if_deps_get_updated() {
+    let fs = InMemoryFileSystem::new();
+    let root = Utf8Path::new("/");
+    let artefact = Utf8Path::new("/artefact");
+
+    // Cache is stale
+    write_src(&fs, "/src/one.gleam", 1, TEST_SOURCE_2);
+    write_cache(&fs, "one", 0, vec![], TEST_SOURCE_1);
+
+    // Cache is fresh but dep is stale
+    write_src(&fs, "/src/two.gleam", 1, "import one");
+    write_cache(
+        &fs,
+        "two",
+        2,
+        vec![(EcoString::from("one"), SrcSpan { start: 0, end: 0 })],
+        "import one",
+    );
+
+    // Cache is fresh
+    write_src(&fs, "/src/three.gleam", 1, TEST_SOURCE_1);
+    write_cache(&fs, "three", 2, vec![], TEST_SOURCE_1);
+
+    let _loaded1 = run_loader(fs.clone(), root, artefact);
+
+    // update the dependency
+    write_cache(&fs, "one", 3, vec![], TEST_SOURCE_2);
+    let loaded2 = run_loader(fs, root, artefact);
+
+    assert_eq!(loaded2.to_compile, vec![EcoString::from("two")]);
+    assert_eq!(
+        loaded2.cached,
+        vec![EcoString::from("one"), EcoString::from("three")]
+    );
+}
+
 #[test]
 fn invalid_module_name() {
     let fs = InMemoryFileSystem::new();