Repository: bitcoin/bitcoin (language: cpp)

Issue:
macOS App Notarization & Stapling
> The Apple notary service is an automated system that scans your software for malicious content, checks for code-signing issues, and returns the results to you quickly. If there are no issues, the notary service generates a ticket for you to staple to your software; the notary service also publishes that ticket online where Gatekeeper can find it.

> Beginning in **macOS 10.14.5**, all new or updated kernel extensions and all software from developers new to distributing with Developer ID must be notarized in order to run. In a future version of macOS, notarization will be required by default for all software.

The currently available version of macOS is 10.14.4.

Apple's notary service requires you to adopt the following protections:
* Enable code-signing for all of the executables you distribute.
* Enable the Hardened Runtime capability for your executable targets, as described in Enable hardened runtime.
* Use a `Developer ID` application, kernel extension, or installer certificate for your code-signing signature.
* Include a secure timestamp with your code-signing signature. (The Xcode distribution workflow includes a secure timestamp by default. For custom workflows, include the `--timestamp` option when running the codesign tool.)
* Don’t include the `com.apple.security.get-task-allow` entitlement with the value set to any variation of true.
* Link against the macOS 10.9 or later SDK.

More info available here: https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution


Discussion / hints:
@jonasschnelli @theuni 
If I'm reading this right, it only applies to "software from developers *new to distributing* with Developer ID", i.e. since you already shipped signed Mac binaries, you should be exempt for the time being. Or were the binaries signed without a Developer ID somehow?
Moving this to `v0.20.0` as we shouldn't have to solve anything here for `v0.19.0`.
I see this message with v0.19 RC3 on MacOS Catalina.

<img width="403" alt="Screen Shot 2019-10-30 at 10 15 40 AM" src="https://user-images.githubusercontent.com/326147/67903298-6992c200-fb28-11e9-9159-a9956662218a.png">

Same issue (macOS 10.15 Catalina).

Note that when copying the DMG from my Gitian machine it doesn't show the same warning. Only when I download it. Files are identical, but presumably macOS tracks this stuff somewhere.
I'm on 10.15 too. I wonder what the difference is?

On Thu, Oct 31, 2019 at 4:18 AM Sjors Provoost <notifications@github.com>
wrote:

> On 10.15 I don't get this message when I update my existing installation
> from the signed DMG, nor when I delete it.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/bitcoin/bitcoin/issues/15774?email_source=notifications&email_token=AACPUAYZQAGRSGET36JZK3DQRK5JNA5CNFSM4HEPXQW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECXLWDQ#issuecomment-548322062>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACPUA6SMQ3O5E4I237DB63QRK5JNANCNFSM4HEPXQWQ>
> .
>

@moneyball none. I updated my comment to reflect that it matters if you get the file as a download or via SSH from a Gitian machine.
> If I'm reading this right, it only applies to "software from developers _new to distributing_ with Developer ID", i.e. since you already shipped signed Mac binaries, you should be exempt for the time being. Or were the binaries signed without a Developer ID somehow?

This is how I too initially understood the documentation. However, as others have pointed out, this does not seem to be the case. With my own project that has distributed code signed binaries for a while now, those same code signed binaries, when downloaded, are actually blocked from running on Catalina with the message shown in @moneyball 's screenshot.

-------------------------------------

Notarization can be done either _after-the-fact_ (post-release) for existing code signed binaries, or during the release process' final steps using `altool` (see https://developer.apple.com/documentation/xcode/notarizing_your_app_before_distribution/customizing_the_notarization_workflow for command examples starting at the "Upload Your App to the Notarization Service" section). This would need to be performed by the code signer.

If all goes well, the binary package will then be recognized by GateKeeper, providing that an active internet connection is available. It is possible to "staple" the notarization receipt to the binary package so that an internet connection is not required, but that would change it's sha256sum, which would no longer match the hashes from gitian's output.

Note: There is an intermittent bug in Apple's notarization system that may block the binary from being notarized properly. Details can be found on Apple's Developer Forums: https://forums.developer.apple.com/thread/118465
> If all goes well, the binary package will then be recognized by GateKeeper, providing that an active internet connection is available.

Which implies giving Apple an overview of all Bitcoin Core macOS user, the exact version they installed and when. Unless GateKeeper downloads a list of hashes, rather than uploading the hash. 

> It is possible to "staple" the notarization receipt to the binary package so that an internet connection is not required, but that would change it's sha256sum, which would no longer match the hashes from gitian's output.

This seems worth some additional hassle for the privacy benefit. During a recent IRC meeting the idea came up to make a tool that strips the extra information from the package and then checks the resulting checksum. 
I had to make sure my app was notarized and "stapled". For those interested, here is what I needed to do to automate the task on travis:

https://github.com/btcpayserver/BTCPayServer.Vault/blob/master/Build/travis/applesign.sh

Basically, I needed to:
* extract the unsigned dmg,
* sign the app,
* rebundle the app as a dmg,
* sign the dmg,
* notarize the dmg,
* wait that it succeed
* Apply the staple to the dmg

Note that applying the staple to the dmg is optional. However, if you don't apply it, gatekeeper will need internet connection to verify the dmg.

I think that I should also **staple the .App folder** itself and not only the dmg if I wanted gatekeeper to be 100% happy when offline. (yes, you can staple the .App folder... I have no idea why it works, where the staple get saved?)
The documentation is very bad, so I don't really know...

If you want to also staple the `.App` I think you need a two steps notarization: First the `.App` folder as a tar file, then the dmg after it has been built.

Take a good note about the parameters I passed to `codesign`, this took a while to get right.
Hi there,
I have just finished signing a **dotnet core application** for distributing app outside the Mac App Store. There are several recommendations by apple and as @NicolasDorier mentioned the documentation to do it outside XCode is very bad quality. 

Take a look at my code: https://github.com/molnard/WalletWasabi/blob/4f499aab964d7c9298a0f041633fb20927d3a18f/WalletWasabi.Packager/MacSignTools.cs

The whole PR: https://github.com/zkSNACKs/WalletWasabi/pull/3041

Oh, just one more thing - I had to move our packager from linux to mac because there is no way to make it work otherwise. 
I looked a bit into this issue and I think a possible approach could be:

* During the manual macOS code signing process via `detached-sig-create.sh` (currently done by me), we could also notarize the .app bundle (and/or the `bitcoin-qt` binary).
* We "staple" the tickets from apple notarisation service directly onto the .app bundle (kinda pack the signature for the notarization into the application bundle so no network connection is required)
* We create the dmg with gitian (pulling in the detached signature & notarization). *This is when the gitian people build the win/osx signed builds.*
* We manually notarize the dmg file (2nd manual step, executed by the macOS code signing key holder)

*(requires testing in order to make sure this process works)*

Downsides:
* A 2nd manual step through macOS code signing maintainer

Alternatives:
* Create directly the dmg in 1st manual code signing step (kinda defeats the purpose of gitian,... so a bad alternative)

Risks:
* Since the final .dmg notarization is a manual one man show step, the question pops up if this could be exploited. I guess it can't. It only tells Apples Gatekeeper server that the gitian built .dmg file was notarized with someone holding the macOS code signing keys. Manually notarizing a malicious .dmg by a maintainer would still require to distribute that malicious .dmg somehow (also codesign).

Anyways, our binary distribution security is based on a completely different layer (deterministic builds with multiple signatures). The os protection with code-signing and notarization is nice but provides much less trust/security IMO.

Ping @theuni @dongcarl.
In practice, given that the macOS code signing maintainer is also a Gitian builder, they can do both manual steps in a single session? 
For test purposes, I notarized the 0.19.1rc**1** dmg file (RC1! not 2!).
If anyone can test if the 0.19.1rc1 gives them a notarization warning ("...apple can not check the app for malicious code"-error should be gone).
I just downloaded it (note: it won't check the notary stuff if you obtain it via SSH or build yourself) and didn't get a notarization warning. Just the usual "you downloaded from bitcoincore.org, are you sure".
Great. So just notarizing the .dmg during the release process would work.
Additionally, we could staple the notarization ticket onto the app bundle. But that would be non trivial via our gitian build process.
"Stapling" would allow detecting the notarization even if no internet connection is available.

I propose that we start with the "we just notarize the dmg"-approach and see if there is demand/need for the "stapling"-approach.
How do users verify the end DMG is generated deterministically by gitian?
@luke-jr it’s still the same process (detached signature). Just an additional flag.
The actual notarization binds to the hash of the deterministic built dmg.
So the notarization is a separate file? How does the end user get that?
@luke-jr 
From my understanding it works like this (simplified):
* We build the dmg in the same was as now
* The macOS codesign key holder notarizes the .dmg file:
* * During notarization, it uploads the .dmg file to apples server and checks its content
* * If the application has all required flags set, etc., it will approve the notarization and stores its hash, etc. on the Apple servers (notarization ticket)
* Once a user starts an application the first time, it checks it's installers hash (the .dmg file) against Apples notarization server and if the hash has an approved notarization ticket, the app launches.

So it needs online connectivity the first time the app launches. Though user can disable the notarization check in the macOS security settings.

It's mandatory since 10.14.5 (means the default security setting only allows notarized apps to launch).

There is an option to add the notarization ticket directly to the app to allow offline users to check the notarization (Apple calls that stapling). But it would complicated our build process. Can be done later.
> * Once a user starts an application the first time, it checks it's installers hash (the .dmg file) against Apples notarization server and if the hash has an approved notarization ticket, the app launches.

This part has me worried. Does Apple use a (provable) anonimzation technique here? Or is this giving them a list of IP address of Bitcoin Core users? Stapling the notarisation onto the DMG seems better, though it makes the process more complicated.
@Sjors: this is indeed a good point. Though I don't know if we can even prevent the ugly online check when we staple the notarization ticket to the application. Maybe it only checks it when the user is offline. But I agree, if we decide to notarize (and I guess we have to), stapling the ticket to the app is the preferred approach.

I just checked what stapling actually does.
It adds a file `CodeResources` to the application bundle (all other files are untouched).
We could add that `CodeResources` file to the detach signature git and "apply" it during gitian-osx-signer process.

The process would look like:
* the macOS code signer needs to run the .dmg creation step (a manual gitian-osx-signer.yml) after he ran the `detached-sig-create.sh`
* ... then, he needs to notarize the created .dmg file
* ... staple the notarization ticket to the .app folder (creates the `CodeResources` file)
* ... commit the detached signatures & the `CodeResources` file to https://github.com/bitcoin-core/bitcoin-detached-sigs
* ... all gitian builders pull the signature & the `CodeResource` file and apply it to the .app folder and create the dmg

Downside:
* More work for the macOS code signer (acceptable IMO)
* Notarization happens before we have proof of determinism and the possibility to check the hashes of other gitian builds (we already have that problem with code signing)
* No current reliable proof if this would not break determinism (I think it wouldn't)
This sounds like a reasonable approach to me.
FWIW, https://sigpipe.macromates.com/2020/macos-catalina-slow-by-design/ "Spawning a new Process" subheader.
At this point this isn't going to happen for 23.0. Moving this to 24.0
It seems that Xcode 13 ships with a new tool called `notarytool` making the whole process easier: https://scriptingosx.com/2021/07/notarize-a-command-line-tool-with-notarytool/

Also the process is supposed to be 10x faster than the old method via `altool` - see https://github.com/electron/electron-notarize/pull/82
might be relevant:
* https://gregoryszorc.com/blog/2022/08/08/achieving-a-completely-open-source-implementation-of-apple-code-signing-and-notarization/
* https://github.com/indygreg/PyOxidizer/tree/main/apple-codesign
* https://gregoryszorc.com/docs/apple-codesign/stable/
* https://crates.io/crates/apple-codesign
But it still phones home and tells Apple you installed Bitcoin Core, right?
Moving this to future. We can revisit depending on what happens during testing of release candidates for 24.x.
> I see this message with v0.19 RC3 on MacOS Catalina.
> 
> <img alt="Screen Shot 2019-10-30 at 10 15 40 AM" width="403" src="https://user-images.githubusercontent.com/326147/67903298-6992c200-fb28-11e9-9159-a9956662218a.png">

I am getting this error message when opening the rc1 24.x
I'm not sure if the release candidates get notarized (and when)?
> I'm not sure if the release candidates get notarized (and when)?

Notarization is not occuring at any point, for any binaries.
Ok, in that case the error message is as expected. You have to right click the first time and then hold `option` when you click `Open`. Needless to say this is an annoying anti open source move by Apple...
> Ok, in that case the error message is as expected. You have to right click the first time and then hold `option` when you click `Open`. Needless to say this is an annoying anti open source move by Apple...

This does not work when a macbook is mdm'ed (corporate and student macbooks). Can we get the rc1 dmg notarized and skip it on the release binaries?
Guys - the user experience is horrible. I download Bitcoin Core from the official webpage - Norton kills my download as it falsely recognizes it as a virus, if I temporarily disable Norton Apple prevents me from opening the downloaded program cause the signing key is outdated and BTC Core is not enabled for multi-key-signing and still depending on the outdated key of the departed Wladimir van der Laan, if I manage to get a valid signing key  and work around Apple preventions Norton kills the running software. This is the future of money? So what is the right way to run BTC Core?
The policy reads to me that notarization will **always** phone home, even in the case of having an externally-stapled ticket, _unless_ a connection is not available in which case the stapled ticket is locally verified. Therefore it seems to me that we have two options available:

1) Notarize the binary and somehow communicate to users that if they don't want Apple to know that they are running Bitcoin Core they must start it up with no internet connection (under the assumption that Gatekeeper only verifies first run).

2) Do not notarize the binary and include instructions similar to those seen on [this page](https://support.apple.com/en-us/HT202491#:~:text=These%20security%20checks%20have%20never%20included%20the%20user%E2%80%99s%20Apple%20ID%20or%20the%20identity%20of%20their%20device.%20To%20further%20protect%20privacy%2C%20we%20have%20stopped%20logging%20IP%20addresses%20associated%20with%20Developer%20ID%20certificate%20checks%2C%20and%20we%20will%20ensure%20that%20any%20collected%20IP%20addresses%20are%20removed%20from%20logs) (copied below) in the MacOS DMG/zip archive, which imply that it is (permanently) saved as an exception.

Personally I'd tend towards the latter option.

> This does not work when a macbook is mdm'ed (corporate and student macbooks). Can we get the rc1 dmg notarized and skip it on the release binaries?

I don't know if option 2 works on an "mdm'ed" macbook though...

----

## If you want to open an app that hasn’t been notarized or is from an unidentified developer

Running software that hasn’t been signed and notarized may expose your computer and personal information to malware that can harm your Mac or compromise your privacy. If you’re certain that an app you want to install is from a trustworthy source and hasn’t been tampered with, you can temporarily override your Mac security settings to open it.

If you still want to open an app for which the developer cannot be verified, open System Preferences.*

<img src="https://user-images.githubusercontent.com/6606587/233847231-ce1eda24-4d06-4687-9a81-da6f326a172b.png" width="250">

Go to Security & Privacy. Click the Open Anyway button in the General pane to confirm your intent to open or install the app. 

<img src="https://user-images.githubusercontent.com/6606587/233847252-02b46f58-f66a-48a6-90ae-415a07210dc1.png" width="450">

The warning prompt reappears, and if you're absolutely sure you want to open the app anyway, you can click Open. 

<img src="https://user-images.githubusercontent.com/6606587/233847256-fae72577-df40-4144-81ed-cdddf0d2183b.png" width="250">

The app is now saved as an exception to your security settings, and you can open it in the future by double-clicking it, just as you can any authorized app. 

Fix patch:
diff --git a/contrib/macdeploy/detached-sig-create.sh b/contrib/macdeploy/detached-sig-create.sh
index 626381cf43027..097a7c35ee451 100755
--- a/contrib/macdeploy/detached-sig-create.sh
+++ b/contrib/macdeploy/detached-sig-create.sh
@@ -24,7 +24,7 @@ fi
 rm -rf ${TEMPDIR}
 mkdir -p ${TEMPDIR}
 
-${SIGNAPPLE} sign -f --detach "${TEMPDIR}/${OUTROOT}"  "$@" "${BUNDLE}"
+${SIGNAPPLE} sign -f --detach "${TEMPDIR}/${OUTROOT}"  "$@" "${BUNDLE}" --hardened-runtime
 
 tar -C "${TEMPDIR}" -czf "${OUT}" .
 rm -rf "${TEMPDIR}"


Test patch: