Repository: tilt-dev/tilt (go)

Issue:
Bug report: Watched file tiltfile reload leaves (Tiltfile) pending until manual reload
### Expected Behavior
When a watched file triggers a tiltfile reload, the tiltfile shouldn't remain in pending after the file(s) is reloaded.

### Current Behavior
On occasion (unclear why), when a watched file triggers a reload of `(Tiltfile)` the tiltfile finishes reloading but the resource remains in pending, which blocks the reload of other services in tilt.

### Steps to Reproduce
1. Create a tiltfile that watches a file, and has resources that depend on the tiltfile (all)
2. Start tilt, let everything settle
3. `touch` the file being watched
4. Note the tiltfile reloads, and will sometimes still be marked as pending after (see image)

![Screen Shot 2022-07-21 at 10 54 00 AM](https://user-images.githubusercontent.com/1282393/180804005-2aae692c-4afa-44ba-b187-723af4ee15e0.png)

### Context
#### `tilt doctor` Output
NOTE: I upgraded to v0.30.5 today and saw this happen again.
<!--- Run `tilt doctor` and paste the output here so we know about your environment -->
```
➜  ~ tilt doctor
Tilt: v0.30.3, built 2022-06-10
System: darwin-amd64
---
Docker
- Host: unix:///var/run/docker.sock
- Server Version: 20.10.17
- API Version: 1.41
- Builder: 2
- Compose Version: v2.6.1
---
Kubernetes
- Env: unknown
- Context: dev-3
- Cluster Name: dev-3
- Namespace: ws-nathan-heaps
- Container Runtime: cri-o
- Version: v1.21.8
- Cluster Local Registry: none
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: opt-out
- Machine: 849283229ca9b586029a6d42301bc3c7
- Repo:
```

#### About Your Use Case
We use tilt heavily, many times >25 services enabled via tilt, and `(Tiltfile)` is all the way at the bottom. This being hit-or-miss tends to trend towards me seeing things hit `Waiting on (Tiltfile)` and not understanding why, if I notice at all.


Pull request description:
fix (Tiltfile) stuck in pending state
Fixes #5902 

I was having the same issue described in #5902 where when using filesystem triggers to add or remove resources, it would sometimes race and leave the Tiltfile in the pending status without the most recent state.

I believe I successfully tracked this down to the filewatch controller when reconciled can replace the watcher object with a new configuration, but doesn't preserve previous file events when doing so. This can then race with the Reconcile on the tiltfile to miss that there is a pending filesystem change and instead see an empty list of filesystem changes (https://github.com/tilt-dev/tilt/blob/59f9aa24c86dc5302772c61905275fd90675596f/internal/controllers/core/tiltfile/reconciler.go#L235-L240). The Tiltfile reconciler then does not trigger a subsequent build.


Patch:
diff --git a/internal/controllers/core/filewatch/controller.go b/internal/controllers/core/filewatch/controller.go
index 13309173c..5b66aedab 100644
--- a/internal/controllers/core/filewatch/controller.go
+++ b/internal/controllers/core/filewatch/controller.go
@@ -190,6 +190,10 @@ func (c *Controller) addOrReplace(ctx context.Context, name types.NamespacedName
 		w.restartBackoff = existing.restartBackoff
 		status.Error = existing.status.Error
 	}
+	if hasExisting {
+		status.FileEvents = existing.status.FileEvents
+		status.LastEventTime = existing.status.LastEventTime
+	}
 
 	ignoreMatcher := ignore.CreateFileChangeFilter(fw.Spec.Ignores)
 	startFileChangeLoop := false


Test patch:
diff --git a/internal/controllers/core/filewatch/controller_test.go b/internal/controllers/core/filewatch/controller_test.go
index ba392f531..1afbe0c6e 100644
--- a/internal/controllers/core/filewatch/controller_test.go
+++ b/internal/controllers/core/filewatch/controller_test.go
@@ -324,13 +324,14 @@ func TestController_Reconcile_Watches(t *testing.T) {
 	assert.Truef(t, updatedStart.After(originalStart),
 		"Monitor start time should be more recent after update, (original: %s, updated: %s)",
 		originalStart, updatedStart)
-	if assert.Equal(t, 2, len(updated.Status.FileEvents)) {
+	if assert.Equal(t, 3, len(updated.Status.FileEvents)) {
 		// ensure ONLY the expected files were seen
 		assert.NotZero(t, updated.Status.FileEvents[0].Time.Time)
-		mostRecentEventTime := updated.Status.FileEvents[1].Time.Time
+		mostRecentEventTime := updated.Status.FileEvents[2].Time.Time
 		assert.NotZero(t, mostRecentEventTime)
-		assert.Equal(t, []string{f.tmpdir.JoinPath("d", "1")}, updated.Status.FileEvents[0].SeenFiles)
-		assert.Equal(t, []string{f.tmpdir.JoinPath("d", "2")}, updated.Status.FileEvents[1].SeenFiles)
+		assert.Equal(t, []string{f.tmpdir.JoinPath("a", "1")}, updated.Status.FileEvents[0].SeenFiles)
+		assert.Equal(t, []string{f.tmpdir.JoinPath("d", "1")}, updated.Status.FileEvents[1].SeenFiles)
+		assert.Equal(t, []string{f.tmpdir.JoinPath("d", "2")}, updated.Status.FileEvents[2].SeenFiles)
 		assert.Equal(t, mostRecentEventTime, updated.Status.LastEventTime.Time)
 	}
 }