Repository: rust-lang/rust-analyzer
Pull request #17025 (2024-04-07 08:43:13)

internal: Use josh for subtree syncs

lnicola: Untested, and still need to update `docs/dev/README.md`.

r? @RalfJung
lnicola: @RalfJung do you know if Josh can use SSH remotes?
lnicola: Doesn't seem to work:

```
Preparing lnicola/rust (base: 688c30dc9f8434d63bddb65bd6a4d2258d19717c)...
$ git fetch https://github.com/rust-lang/rust 688c30dc9f8434d63bddb65bd6a4d2258d19717c
From https://github.com/rust-lang/rust
 * branch                    688c30dc9f8434d63bddb65bd6a4d2258d19717c -> FETCH_HEAD
$ git push https://github.com/lnicola/rust 688c30dc9f8434d63bddb65bd6a4d2258d19717c:refs/heads/sync-from-ra

Pushing rust-analyzer changes...
$ git push http://localhost:42042/lnicola/rust.git:rev(f5a9250147f6569d8d89334dc9cca79c0322729f:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer.git HEAD:sync-from-ra
To http://localhost:42042/lnicola/rust.git:rev(f5a9250147f6569d8d89334dc9cca79c0322729f:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer.git
 ! [rejected]              HEAD -> sync-from-ra (non-fast-forward)
error: failed to push some refs to 'http://localhost:42042/lnicola/rust.git:rev(f5a9250147f6569d8d89334dc9cca79c0322729f:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer.git'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. If you want to integrate the remote changes, use 'git pull'
hint: before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```

Sounds like I didn't delete the branch on my fork, but I did.
RalfJung: I'm using josh via SSH, it's working fine for me. Also see the git config hint at the end of [this section](https://github.com/rust-lang/miri/blob/master/CONTRIBUTING.md#exporting-changes-to-the-rustc-repo).
RalfJung: It's hard to review this. The code seems to be mostly exactly like in Miri. So you'll just have to try it. First try pulling, and see if it produces only a few new commits compared to RA master. Opening a PR is a good way to check that as github lists all new commits. Then try pushing and see that that also only produces a few new commits. Then try changing something in the "pulled" branch and try pushing *that*, and try changing something in the pushed branch and try pulling that.
lnicola: > So you'll just have to try it.

Well, I did. Pulling doesn't produce any commits, which is normal. But pushing fails as above.
RalfJung: Hm, either the rust-version file needs to use a different commit then or the first push is magic and needs to be done by hand somehow... I forgot what exactly we did initially in Miri to get this started.

What if you put the current rustc HEAD into the rust-version file?

Are you sure pulling from the current rustc HEAD produces no new commits? There are usually some merge commits being added. It may be that you can only push after those have been integrated.
lnicola: Yes, I also tried that. Maybe we should do a pull first?
RalfJung: Yeah try doing the push on a branch where a pull already happened. That also avoids having to hand craft that rust-version file.
lnicola: Hmm, interesting:

 - with the `-git` version of `josh`, pull is also broken:

```
$ git fetch http://localhost:42042/rust-lang/rust.git@fc1a4c5cc9308c4b5980c64a73fd344a59c10601:rev(f5a9250147f6569d8d89334dc9cca79c0322729f:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer.git
fatal: couldn't find remote ref HEAD
```

 - I tried to pull on this branch, but:

```
Your branch is ahead of 'mine/josh' by 1639 commits.
  (use "git push" to publish your local commits)
```
RalfJung: That's a lot of commits, but they seem to all be merge commits? Every merge commit in rustc that spans an RA change will show up here, this is needed for josh to be able to match the repo structures against each other.

However, that still seems like too many commits. I don't think we got 1600 commits in Miri when we did the initial josh sync.
lnicola: Oh, I hate this:

```
$ git reset d28e67a35f
Unstaged changes after reset:
M	rust-version
fatal: sha1 file '.git/index.lock' write error. Out of diskspace
```

Besides the space needed for the rustc checkout for push (miri-script says 1 GB, but it's about 13 GB for me?), josh keeps another copy and a half of the repo in its cache:

```
9,8G	~/.cache/rust-analyzer-josh/
```
RalfJung: I can reproduce the strange result for rustc-pull, where it generates an enormous amount of merge commits. This is not something we saw when switching Miri to Josh.

I wonder if it's related to the fact that the git subtree syncs here look different than what I remember from Miri, with these huge commits created by rustc pushes.

I spot-checked some of these commits and they all make sense, in a sense. For instance, `Auto merge of #3145 - RalfJung:data-race-error, r=RalfJung` shows up as 4279268cb09daeca33a3b893aa0680633d9bdfd2; in rustc that's 2ef639d4fbf89dc584bfa1a2211b56123df4aa5f with the to parent commits 1b2e4a9c941, 278965a0c49. (This is in fact originally a Miri commit.) `git diff 1b2e4a9c941 278965a0c49 src/tools/rust-analyzer` shows that indeed there is a diff here, so for josh to preserve the history it has to include this merge commit in the repo. The diff is the same as that of the two parent commits of the corresponding commit on the RA side. When doing rustc-pull, josh will preserve all merge commits whose parents differ on the parts of the tree that josh is currently fetching.
RalfJung: However, pushing in the other direction seems to work just right for me. I did a pull, and then I changed something in the readme and did a push, and the result is [this branch](https://github.com/RalfJung/rust/tree/sync-from-ra) which does round-trip properly and which only adds 4 commits to rustc.

I then tried changing something there and pulling that back into RA, which also worked fine.

Not sure what went wrong when your push ended up non-roundtripping. If you pushed on a branch that didn't previously get a rustc-pull then that is probably the reason; anything you try to push should be based on what josh produces when pulling so that josh has a chance of matching up the histories and reconstructing a suitable RA history on the rustc side.
lnicola: I guess I must be doing something really wrong then. Here are my steps:

 - I'm on a clean checkout of this branch (`lnicola:josh`)
 - I do a `cargo xtask rustc-pull`, which only adds two commits:

```
commit 46b460d8c961508f580bd05e5d168e7bb66722bb (HEAD -> josh)
Merge: e8690219a0 ae00688f19
Author: Laurențiu Nicola <lnicola@dend.ro>
Date:   Sat Apr 20 09:51:41 2024 +0300

    Merge from downstream

commit e8690219a01df5451ff29e7853cf645ac3003f93
Author: Laurențiu Nicola <lnicola@dend.ro>
Date:   Sat Apr 20 09:51:32 2024 +0300

    Preparing for merge from downstream

diff --git a/rust-version b/rust-version
index e2a22b2395..e1d07306ff 100644
--- a/rust-version
+++ b/rust-version
@@ -1 +1 @@
-688c30dc9f8434d63bddb65bd6a4d2258d19717c
+9c7b1f4848263b7a39486a2cd482db0d632884e8
```

 - I have a clean `rust-lang/rust` checkout at `../rust-rust-analyzer`, currently at `9c7b1f4848263b7a39486a2cd482db0d632884e8`
 - I run `RUSTC_GIT=../rust-rust-analyzer GITHUB_USER=lnicola cargo xtask rustc-push`, which pushes to my branch at https://github.com/lnicola/rust/tree/sync-from-ra, but fails:

```
Preparing lnicola/rust (base: c8d19a92aa9022eb690899cf6d54fd23cb6877e5)...
$ git fetch https://github.com/rust-lang/rust c8d19a92aa9022eb690899cf6d54fd23cb6877e5
From https://github.com/rust-lang/rust
 * branch                    c8d19a92aa9022eb690899cf6d54fd23cb6877e5 -> FETCH_HEAD
$ git push https://github.com/lnicola/rust c8d19a92aa9022eb690899cf6d54fd23cb6877e5:refs/heads/sync-from-ra

Pushing rust-analyzer changes...
$ git push http://localhost:42042/lnicola/rust.git:rev(f5a9250147f6569d8d89334dc9cca79c0322729f:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer.git HEAD:sync-from-ra
Enumerating objects: 4271, done.
Counting objects: 100% (2468/2468), done.
Delta compression using up to 32 threads
Compressing objects: 100% (671/671), done.
Writing objects: 100% (1571/1571), 325.51 KiB | 46.50 MiB/s, done.
Total 1571 (delta 1036), reused 1368 (delta 856), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (1036/1036), completed with 285 local objects.
remote: josh-proxy
remote: response from upstream:
remote: To https://github.com/lnicola/rust.git
remote:    c8d19a92aa9..390772a056a  JOSH_PUSH -> sync-from-ra
remote: REWRITE(5c6a075f170ebcdaef5aadde3d29f80d6f4f6356 -> 7980fac7b164141a978f89fae3b7dd349787e02a)
remote: 
remote: 
To http://localhost:42042/lnicola/rust.git:rev(f5a9250147f6569d8d89334dc9cca79c0322729f:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer.git
   f56ffd45ed..5c6a075f17  HEAD -> sync-from-ra

Error: Josh created a non-roundtrip push! Do NOT merge this into rustc!
```

 - interestingly, GitHub says: This branch is [157 commits ahead of](https://github.com/lnicola/rust/compare/rust-lang%3Arust%3Amaster...sync-from-ra), [2 commits behind](https://github.com/lnicola/rust/compare/sync-from-ra...rust-lang%3Arust%3Amaster) rust-lang/rust:master. I'm not sure what's up with those two commits.
RalfJung: > I do a cargo xtask rustc-pull, which only adds two commits:

That is extremely strange. How did you determine that it only adds two commits? When I look at the history of my branch after doing this, I see a lot of new commits -- and you also saw these at least once, since at some point you pushed to this PR a branch that had all these commits.

FWIW my version of josh is
```
RUSTFLAGS="--cap-lints=warn" cargo +stable install josh-proxy --git https://github.com/josh-project/josh --tag r23.12.04
```

I have pushed the result of running `cargo xtask rustc-pull` on commit 2228677b93 to https://github.com/RalfJung/rust-analyzer/tree/josh.
lnicola: > That is extremely strange. How did you determine that it only adds two commits? When I look at the history of my branch after doing this, I see a lot of new commits -- and you also saw these at least once, since at some point you pushed to this PR a branch that had all these commits.

You're right, I didn't pay attention.

> FWIW my version of josh is

I have the same one. When I tried `-git`, it failed on pulls.

> Something is seriously borked... maybe try clearing your josh cache?

It was fresh, I just reinstalled my OS.
lnicola: > I have pushed the result of running cargo xtask rustc-pull on commit https://github.com/rust-lang/rust-analyzer/commit/2228677b93b998d9e7ec9da5d8c09a1728670420 to https://github.com/RalfJung/rust-analyzer/tree/josh. It's a ton of commits, but the diff is not very big

This is mine: https://github.com/lnicola/rust-analyzer/tree/josh-plus-rustc-pull
RalfJung: > I have the same one. When I tried -git, it failed on pulls.

This is from `--git`. Do you mean, when you tried without the `--tag` it fails on pulls? I'll see if I can reproduce this.

> This is mine: https://github.com/lnicola/rust-analyzer/tree/josh-plus-rustc-pull

`git diff` between our two branches shows no difference. But your `git diff` above looks different from mine. Is that diff wrong?
lnicola: > Do you mean, when you tried without the --tag it fails on pulls?

Yes.

> But your git diff above looks different from mine, it doesn't have the change in lexed_str.rs. Or is that not a full diff?

Sorry for misleading you, that was just the start of `git log`, not a diff.

```
$ git diff josh-plus-rustc-pull..mine/josh # this is lnicola/josh
crates/parser/src/lexed_str.rs --- Rust
178                 rustc_lexer::TokenKind::Ident => {
179                     SyntaxKind::from_keyword(token_text).unwrap_or(IDENT)
180                 }
181                 rustc_lexer::TokenKind::InvalidPrefix | rustc_lexer::TokenKind::InvalidIdent => {
182                     err = "Ident contains invalid characters";
183                     IDENT
184                 }

rust-version --- 1/2 --- Text
1 9c7b1f4848263b7a39486a2cd482db0d632884e8

rust-version --- 2/2 --- Text
1 688c30dc9f8434d63bddb65bd6a4d2258d19717c
```

RalfJung: > > Do you mean, when you tried without the --tag it fails on pulls?
> 
> Yes.

I can't reproduce this. I just tried `RUSTFLAGS="--cap-lints=warn" cargo +stable install josh-proxy --git https://github.com/josh-project/josh` and then reset my branch back to 2228677 and run `cargo xtask rustc-pull` again and it succeeds just fine, producing a branch identical in content (but with a different hash) to the one I got with the tag.
RalfJung: I went back to the josh tag, to get a stable version.

> I run RUSTC_GIT=../rust-rust-analyzer GITHUB_USER=lnicola cargo xtask rustc-push, which pushes to my branch at https://github.com/lnicola/rust/tree/sync-from-ra, but fails:

Interesting, I also get this when I try your branch. Though the message doesn't quite align -- yours says something about 5c6a075f17 where I would expect c0f9fa7f53; is that output from a previous attempt?

And now I even get such a non-roundtrip push with my own branch that I just created. That is strange as it worked fine last time.
RalfJung: The same happens when I do `cargo xtask rustc-pull --commit 84e729a59f216cc64755788a470f165429a731f4` (just to fix another variable, namely the moving HEAD of rustc).

What changed between 7279deaa55ff64e4c7f72752ad0d13bd1e315612 and 2228677?
When I go back to 7279deaa55ff64e4c7f72752ad0d13bd1e315612 and do  `cargo xtask rustc-pull --commit 84e729a59f216cc64755788a470f165429a731f4`  and then a push, that works fine.

lnicola: Nothing, AFAICT. I just rebased today on top of `master`, but I had the same issue on 56021a0d6123ced47c4db622f5b7f56a5f9ae97f.
RalfJung: > Nothing, AFAICT. I just rebased today on top of `master`, but I had the same issue on [56021a0](https://github.com/rust-lang/rust-analyzer/commit/56021a0d6123ced47c4db622f5b7f56a5f9ae97f).

I suspect #16983 has something to do with it. josh doesn't seem able to push subtree syncs.

So it is probably necessary to do a git subtree sync into rustc, and then update the josh filter to use *that* synced commit as the base, instead of using f5a9250147f6569d8d89334dc9cca79c0322729f.
RalfJung: I have written down what we learned about migrating to josh [in a hackmd](https://hackmd.io/7pOuxnkdQDaL1Y1FQr65xg), so maybe it's easier next time. ;)

Patch:
diff --git a/Cargo.lock b/Cargo.lock
index a6e460134f..bd3894b4d4 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -323,6 +323,27 @@ dependencies = [
  "syn",
 ]
 
+[[package]]
+name = "directories"
+version = "5.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9a49173b84e034382284f27f1af4dcbbd231ffa358c0fe316541a7337f376a35"
+dependencies = [
+ "dirs-sys",
+]
+
+[[package]]
+name = "dirs-sys"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "520f05a5cbd335fae5a99ff7a6ab8627577660ee5cfd6a94a6a929b52ff0321c"
+dependencies = [
+ "libc",
+ "option-ext",
+ "redox_users",
+ "windows-sys 0.48.0",
+]
+
 [[package]]
 name = "dissimilar"
 version = "1.0.7"
@@ -898,6 +919,16 @@ dependencies = [
  "libc",
 ]
 
+[[package]]
+name = "libredox"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c0ff37bd590ca25063e35af745c343cb7a0271906fb7b37e4813e8f79f00268d"
+dependencies = [
+ "bitflags 2.4.2",
+ "libc",
+]
+
 [[package]]
 name = "limit"
 version = "0.0.0"
@@ -1186,6 +1217,12 @@ version = "11.1.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0ab1bc2a289d34bd04a330323ac98a1b4bc82c9d9fcb1e66b63caa84da26b575"
 
+[[package]]
+name = "option-ext"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
+
 [[package]]
 name = "parking_lot"
 version = "0.12.1"
@@ -1566,6 +1603,17 @@ dependencies = [
  "bitflags 1.3.2",
 ]
 
+[[package]]
+name = "redox_users"
+version = "0.4.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bd283d9651eeda4b2a83a43c1c91b266c40fd76ecd39a50a8c630ae69dc72891"
+dependencies = [
+ "getrandom",
+ "libredox",
+ "thiserror",
+]
+
 [[package]]
 name = "rowan"
 version = "0.15.15"
@@ -2499,6 +2547,7 @@ name = "xtask"
 version = "0.1.0"
 dependencies = [
  "anyhow",
+ "directories",
  "flate2",
  "itertools",
  "proc-macro2",
diff --git a/docs/dev/README.md b/docs/dev/README.md
index cdab6b0992..8897f02e27 100644
--- a/docs/dev/README.md
+++ b/docs/dev/README.md
@@ -229,12 +229,22 @@ Release steps:
      * publishes the VS Code extension to the marketplace
    * call the GitHub API for PR details
    * create a new changelog in `rust-analyzer.github.io`
-3. While the release is in progress, fill in the changelog
-4. Commit & push the changelog
+3. While the release is in progress, fill in the changelog.
+4. Commit & push the changelog.
 5. Run `cargo xtask publish-release-notes <CHANGELOG>` -- this will convert the changelog entry in AsciiDoc to Markdown and update the body of GitHub Releases entry.
-6. Tweet
-7. Inside `rust-analyzer`, run `cargo xtask promote` -- this will create a PR to rust-lang/rust updating rust-analyzer's subtree.
-   Self-approve the PR.
+6. Tweet.
+7. Make a new branch and run `cargo xtask rustc-pull`, open a PR, and merge it.
+   This will pull any changes from `rust-lang/rust` into `rust-analyzer`.
+8. Switch to `master`, pull, then run `cargo xtask rustc-push --rust-path ../rust-rust-analyzer --rust-fork matklad/rust`.
+   Replace `matklad/rust` with your own fork of `rust-lang/rust`.
+   You can use the token to authenticate when you get prompted for a password, since `josh` will push over HTTPS, not SSH.
+   This will push the `rust-analyzer` changes to your fork.
+   You can then open a PR against `rust-lang/rust`.
+
+Note: besides the `rust-rust-analyzer` clone, the Josh cache (stored under `~/.cache/rust-analyzer-josh`) will contain a bare clone of `rust-lang/rust`.
+This currently takes about 3.5 GB.
+
+This [HackMD](https://hackmd.io/7pOuxnkdQDaL1Y1FQr65xg) has details about how `josh` syncs work.
 
 If the GitHub Actions release fails because of a transient problem like a timeout, you can re-run the job from the Actions console.
 If it fails because of something that needs to be fixed, remove the release tag (if needed), fix the problem, then start over.
diff --git a/rust-version b/rust-version
new file mode 100644
index 0000000000..e2a22b2395
--- /dev/null
+++ b/rust-version
@@ -0,0 +1,1 @@
+688c30dc9f8434d63bddb65bd6a4d2258d19717c
diff --git a/xtask/Cargo.toml b/xtask/Cargo.toml
index a83d32e414..192de86947 100644
--- a/xtask/Cargo.toml
+++ b/xtask/Cargo.toml
@@ -8,6 +8,7 @@ rust-version.workspace = true
 
 [dependencies]
 anyhow.workspace = true
+directories = "5.0"
 flate2 = "1.0.24"
 write-json = "0.1.2"
 xshell.workspace = true
diff --git a/xtask/src/flags.rs b/xtask/src/flags.rs
index 9066545920..dd7bfd0bda 100644
--- a/xtask/src/flags.rs
+++ b/xtask/src/flags.rs
@@ -14,16 +14,16 @@ xflags::xflags! {
         cmd install {
             /// Install only VS Code plugin.
             optional --client
-            /// One of 'code', 'code-exploration', 'code-insiders', 'codium', or 'code-oss'.
+            /// One of `code`, `code-exploration`, `code-insiders`, `codium`, or `code-oss`.
             optional --code-bin name: String
 
             /// Install only the language server.
             optional --server
-            /// Use mimalloc allocator for server
+            /// Use mimalloc allocator for server.
             optional --mimalloc
-            /// Use jemalloc allocator for server
+            /// Use jemalloc allocator for server.
             optional --jemalloc
-            /// build in release with debug info set to 2
+            /// build in release with debug info set to 2.
             optional --dev-rel
         }
 
@@ -32,9 +32,21 @@ xflags::xflags! {
         cmd release {
             optional --dry-run
         }
-        cmd promote {
-            optional --dry-run
+
+        cmd rustc-pull {
+            /// rustc commit to pull.
+            optional --commit refspec: String
+        }
+
+        cmd rustc-push {
+            /// rust local path, e.g. `../rust-rust-analyzer`.
+            required --rust-path rust_path: String
+            /// rust fork name, e.g.  `matklad/rust`.
+            required --rust-fork rust_fork: String
+            /// branch name.
+            optional --branch branch: String
         }
+
         cmd dist {
             /// Use mimalloc allocator for server
             optional --mimalloc
@@ -77,7 +89,8 @@ pub enum XtaskCmd {
     Install(Install),
     FuzzTests(FuzzTests),
     Release(Release),
-    Promote(Promote),
+    RustcPull(RustcPull),
+    RustcPush(RustcPush),
     Dist(Dist),
     PublishReleaseNotes(PublishReleaseNotes),
     Metrics(Metrics),
@@ -104,8 +117,15 @@ pub struct Release {
 }
 
 #[derive(Debug)]
-pub struct Promote {
-    pub dry_run: bool,
+pub struct RustcPull {
+    pub commit: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct RustcPush {
+    pub rust_path: String,
+    pub rust_fork: String,
+    pub branch: Option<String>,
 }
 
 #[derive(Debug)]
diff --git a/xtask/src/main.rs b/xtask/src/main.rs
index 9418675a34..e070576303 100644
--- a/xtask/src/main.rs
+++ b/xtask/src/main.rs
@@ -34,7 +34,8 @@ fn main() -> anyhow::Result<()> {
         flags::XtaskCmd::Install(cmd) => cmd.run(sh),
         flags::XtaskCmd::FuzzTests(_) => run_fuzzer(sh),
         flags::XtaskCmd::Release(cmd) => cmd.run(sh),
-        flags::XtaskCmd::Promote(cmd) => cmd.run(sh),
+        flags::XtaskCmd::RustcPull(cmd) => cmd.run(sh),
+        flags::XtaskCmd::RustcPush(cmd) => cmd.run(sh),
         flags::XtaskCmd::Dist(cmd) => cmd.run(sh),
         flags::XtaskCmd::PublishReleaseNotes(cmd) => cmd.run(sh),
         flags::XtaskCmd::Metrics(cmd) => cmd.run(sh),
diff --git a/xtask/src/release.rs b/xtask/src/release.rs
index 1a5e6dfb4c..9dcf7af00b 100644
--- a/xtask/src/release.rs
+++ b/xtask/src/release.rs
@@ -1,5 +1,12 @@
 mod changelog;
 
+use std::process::{Command, Stdio};
+use std::thread;
+use std::time::Duration;
+
+use anyhow::{bail, Context as _};
+use directories::ProjectDirs;
+use stdx::JodChild;
 use xshell::{cmd, Shell};
 
 use crate::{codegen, date_iso, flags, is_release_tag, project_root};
@@ -71,26 +78,154 @@ impl flags::Release {
     }
 }
 
-impl flags::Promote {
+// git sync implementation adapted from https://github.com/rust-lang/miri/blob/62039ac/miri-script/src/commands.rs
+impl flags::RustcPull {
     pub(crate) fn run(self, sh: &Shell) -> anyhow::Result<()> {
-        let _dir = sh.push_dir("../rust-rust-analyzer");
-        cmd!(sh, "git switch master").run()?;
-        cmd!(sh, "git fetch upstream").run()?;
-        cmd!(sh, "git reset --hard upstream/master").run()?;
+        sh.change_dir(project_root());
+        let commit = self.commit.map(Result::Ok).unwrap_or_else(|| {
+            let rust_repo_head =
+                cmd!(sh, "git ls-remote https://github.com/rust-lang/rust/ HEAD").read()?;
+            rust_repo_head
+                .split_whitespace()
+                .next()
+                .map(|front| front.trim().to_owned())
+                .ok_or_else(|| anyhow::format_err!("Could not obtain Rust repo HEAD from remote."))
+        })?;
+        // Make sure the repo is clean.
+        if !cmd!(sh, "git status --untracked-files=no --porcelain").read()?.is_empty() {
+            bail!("working directory must be clean before running `cargo xtask pull`");
+        }
+        // Make sure josh is running.
+        let josh = start_josh()?;
 
-        let date = date_iso(sh)?;
-        let branch = format!("rust-analyzer-{date}");
-        cmd!(sh, "git switch -c {branch}").run()?;
-        cmd!(sh, "git subtree pull -m ':arrow_up: rust-analyzer' -P src/tools/rust-analyzer rust-analyzer release").run()?;
+        // Update rust-version file. As a separate commit, since making it part of
+        // the merge has confused the heck out of josh in the past.
+        // We pass `--no-verify` to avoid running any git hooks that might exist,
+        // in case they dirty the repository.
+        sh.write_file("rust-version", format!("{commit}\n"))?;
+        const PREPARING_COMMIT_MESSAGE: &str = "Preparing for merge from rust-lang/rust";
+        cmd!(sh, "git commit rust-version --no-verify -m {PREPARING_COMMIT_MESSAGE}")
+            .run()
+            .context("FAILED to commit rust-version file, something went wrong")?;
 
-        if !self.dry_run {
-            cmd!(sh, "git push -u origin {branch}").run()?;
-            cmd!(
-                sh,
-                "xdg-open https://github.com/matklad/rust/pull/new/{branch}?body=r%3F%20%40ghost"
-            )
+        // Fetch given rustc commit.
+        cmd!(sh, "git fetch http://localhost:{JOSH_PORT}/rust-lang/rust.git@{commit}{JOSH_FILTER}.git")
+            .run()
+            .map_err(|e| {
+                // Try to un-do the previous `git commit`, to leave the repo in the state we found it it.
+                cmd!(sh, "git reset --hard HEAD^")
+                    .run()
+                    .expect("FAILED to clean up again after failed `git fetch`, sorry for that");
+                e
+            })
+            .context("FAILED to fetch new commits, something went wrong (committing the rust-version file has been undone)")?;
+
+        // Merge the fetched commit.
+        const MERGE_COMMIT_MESSAGE: &str = "Merge from rust-lang/rust";
+        cmd!(sh, "git merge FETCH_HEAD --no-verify --no-ff -m {MERGE_COMMIT_MESSAGE}")
+            .run()
+            .context("FAILED to merge new commits, something went wrong")?;
+
+        drop(josh);
+        Ok(())
+    }
+}
+
+impl flags::RustcPush {
+    pub(crate) fn run(self, sh: &Shell) -> anyhow::Result<()> {
+        let branch = self.branch.as_deref().unwrap_or("sync-from-ra");
+        let rust_path = self.rust_path;
+        let rust_fork = self.rust_fork;
+
+        sh.change_dir(project_root());
+        let base = sh.read_file("rust-version")?.trim().to_owned();
+        // Make sure the repo is clean.
+        if !cmd!(sh, "git status --untracked-files=no --porcelain").read()?.is_empty() {
+            bail!("working directory must be clean before running `cargo xtask push`");
+        }
+        // Make sure josh is running.
+        let josh = start_josh()?;
+
+        // Find a repo we can do our preparation in.
+        sh.change_dir(rust_path);
+
+        // Prepare the branch. Pushing works much better if we use as base exactly
+        // the commit that we pulled from last time, so we use the `rust-version`
+        // file to find out which commit that would be.
+        println!("Preparing {rust_fork} (base: {base})...");
+        if cmd!(sh, "git fetch https://github.com/{rust_fork} {branch}")
+            .ignore_stderr()
+            .read()
+            .is_ok()
+        {
+            bail!(
+                "The branch `{branch}` seems to already exist in `https://github.com/{rust_fork}`. Please delete it and try again."
+            );
+        }
+        cmd!(sh, "git fetch https://github.com/rust-lang/rust {base}").run()?;
+        cmd!(sh, "git push https://github.com/{rust_fork} {base}:refs/heads/{branch}")
+            .ignore_stdout()
+            .ignore_stderr() // silence the "create GitHub PR" message
             .run()?;
+        println!();
+
+        // Do the actual push.
+        sh.change_dir(project_root());
+        println!("Pushing rust-analyzer changes...");
+        cmd!(
+            sh,
+            "git push http://localhost:{JOSH_PORT}/{rust_fork}.git{JOSH_FILTER}.git HEAD:{branch}"
+        )
+        .run()?;
+        println!();
+
+        // Do a round-trip check to make sure the push worked as expected.
+        cmd!(
+            sh,
+            "git fetch http://localhost:{JOSH_PORT}/{rust_fork}.git{JOSH_FILTER}.git {branch}"
+        )
+        .ignore_stderr()
+        .read()?;
+        let head = cmd!(sh, "git rev-parse HEAD").read()?;
+        let fetch_head = cmd!(sh, "git rev-parse FETCH_HEAD").read()?;
+        if head != fetch_head {
+            bail!("Josh created a non-roundtrip push! Do NOT merge this into rustc!");
         }
+        println!("Confirmed that the push round-trips back to rust-analyzer properly. Please create a rustc PR:");
+        // https://github.com/github-linguist/linguist/compare/master...octocat:linguist:master
+        let fork_path = rust_fork.replace('/', ":");
+        println!(
+            "    https://github.com/rust-lang/rust/compare/{fork_path}:{branch}?quick_pull=1&title=Subtree+update+of+rust-analyzer&body=r?+@ghost"
+        );
+
+        drop(josh);
         Ok(())
     }
 }
+
+/// Used for rustc syncs.
+const JOSH_FILTER: &str =
+    ":rev(55d9a533b309119c8acd13061581b43ae8840823:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer";
+const JOSH_PORT: &str = "42042";
+
+fn start_josh() -> anyhow::Result<impl Drop> {
+    // Determine cache directory.
+    let local_dir = {
+        let user_dirs = ProjectDirs::from("org", "rust-lang", "rust-analyzer-josh").unwrap();
+        user_dirs.cache_dir().to_owned()
+    };
+
+    // Start josh, silencing its output.
+    let mut cmd = Command::new("josh-proxy");
+    cmd.arg("--local").arg(local_dir);
+    cmd.arg("--remote").arg("https://github.com");
+    cmd.arg("--port").arg(JOSH_PORT);
+    cmd.arg("--no-background");
+    cmd.stdout(Stdio::null());
+    cmd.stderr(Stdio::null());
+    let josh = cmd.spawn().context("failed to start josh-proxy, make sure it is installed")?;
+    // Give it some time so hopefully the port is open. (100ms was not enough.)
+    thread::sleep(Duration::from_millis(200));
+
+    Ok(JodChild(josh))
+}


Test patch:
diff --git a/crates/rust-analyzer/tests/slow-tests/tidy.rs b/crates/rust-analyzer/tests/slow-tests/tidy.rs
index 3443939133..4a7415b016 100644
--- a/crates/rust-analyzer/tests/slow-tests/tidy.rs
+++ b/crates/rust-analyzer/tests/slow-tests/tidy.rs
@@ -144,6 +144,7 @@ MIT OR Apache-2.0
 MIT OR Apache-2.0 OR Zlib
 MIT OR Zlib OR Apache-2.0
 MIT/Apache-2.0
+MPL-2.0
 Unlicense OR MIT
 Unlicense/MIT
 Zlib OR Apache-2.0 OR MIT