Repository: RustPython/RustPython (language: rust)

Issue:
Failed cargo install with SSL feature without openssl3
## Summary

1. Issued command `cargo install --git https://github.com/RustPython/RustPython --features=ssl`
2. Got error while building

## System Info

* Linux: Ubuntu 20.04 LTS
* Rust: rustc 1.76.0 (07dca489a 2024-02-04)
* RustPython: 0.3.0
* Openssl: openssl (1.1.1f-1ubuntu2.22)
* `libssl-dev` (1.1.1f-1ubuntu2.22)

> Need more info? Just ask.

## Actual

```
    Updating git repository `https://github.com/RustPython/RustPython`
  Installing rustpython v0.3.0 (https://github.com/RustPython/RustPython#e6c6f966)
    Updating crates.io index
    Updating git repository `https://github.com/RustPython/Parser.git`
    Updating git repository `https://github.com/RustPython/__doc__`
  Downloaded log v0.4.21
  Downloaded syn v2.0.52
  Downloaded 2 crates (298.2 KB) in 3.34s
   Compiling version_check v0.9.4
   ( ... Shortened the long list of Downloaded and Compiling ... )
error[E0599]: no method named `all_certificates` found for reference `&X509StoreRef` in the current scope
   --> stdlib/src/ssl.rs:706:18
    |
703 |               let certs = self
    |  _________________________-
704 | |                 .ctx()
705 | |                 .cert_store()
706 | |                 .all_certificates()
    | |                 -^^^^^^^^^^^^^^^^ method not found in `&X509StoreRef`
    | |_________________|
    | 

For more information about this error, try `rustc --explain E0599`.
error: could not compile `rustpython-stdlib` (lib) due to 1 previous error
error: failed to compile `rustpython v0.3.0 (https://github.com/RustPython/RustPython#e6c6f966)`, intermediate artifacts can be found at `/tmp/cargo-installm6J4G5`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
```

Things I've done to try to fix:

- Checked `rustup update` for rust updates (`rustc 1.76.0 (07dca489a 2024-02-04)`)
- Tried `cargo install --git https://github.com/RustPython/RustPython` (non-SSL works fine)
- Tried [d0c827ed] with SSL (no luck)

> If anyone could offer further suggestions.


Discussion / hints:
Did you install the `libssl-dev` package?
> Did you install the `libssl-dev` package?

`libssl-dev` (1.1.1f-1ubuntu2.22)

> Updated initial comment (added libssl-dev version and added attempt with [d0c827e])
I have exact same issue on AlmaLinux release 8.9 (Midnight Oncilla).
`openssl-devel-1.1.1k-9.el8_7.x86_64` is installed.
Same on Windows with open-ssl installed.
![image](https://github.com/RustPython/RustPython/assets/11755312/55e1403e-cf8e-47a6-ad7d-35c53e704ff2)

Have you solved the problem?
> I have exact same issue on AlmaLinux release 8.9 (Midnight Oncilla). `openssl-devel-1.1.1k-9.el8_7.x86_64` is installed.
Have you solved the problem?

Encountered the same issue with [aws-lc](https://github.com/aws/aws-lc). It is also based on openssl 1.1.x so maybe that has something to do with it?

Edit: Reverting the changes from https://github.com/RustPython/RustPython/commit/32f662ae806848dd4fad7c38dc5398636b6222f0 in stdlib/src/ssl seems to fix it, however I ran into a bunch of other errors during link time 👀
![image](https://github.com/RustPython/RustPython/assets/11761863/ce99719d-7b4d-499c-86b4-94d957be3919)

Yeah, I suspect openssl 1.1.x is a culprit since I can build without an issue on Fedora 39 (openssl 3.1.1) and Ubutnu 22.04 (openssl 3.0.2).
> Yeah, I suspect openssl 1.1.x is a culprit since I can build without an issue on Fedora 39 (openssl

You are right. I consulted with project contributor, and his anwer was the same as yours.
Thank you for confirming. How about using `--features ssl-vendor`? Does it help to solve the problem?
Thanks! `--features ssl-vendor` works on AlmaLinux 8.9

On Windows, it appears we need to install `perl` separately

```
error: failed to run custom build command for `openssl-sys v0.9.98`

Caused by:
  process didn't exit successfully: `C:\msys64\home\ryuta\packages\rustpython\rustpython\target\release\build\openssl-sys-0e5ebd593dca87ba\build-script-main` (exit code: 101)
  --- stdout
  cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_OPENSSL_NO_VENDOR
  X86_64_PC_WINDOWS_MSVC_OPENSSL_NO_VENDOR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_VENDOR
  OPENSSL_NO_VENDOR unset
  running "perl" "./Configure" "--prefix=C:/msys64/home/ryuta/packages/rustpython/rustpython/target/release/build/openssl-sys-d6016873a007658f/out/openssl-build/install" "--openssldir=SYS$MANAGER:[OPENSSL]" "no-dso" "no-shared" "no-ssl3" "no-tests" "no-comp" "no-zlib" "no-zlib-dynamic" "--libdir=lib" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "no-capieng" "no-asm" "VC-WIN64A"

  --- stderr
  thread 'main' panicked at C:\Users\ryuta\.cargo\registry\src\index.crates.io-6f17d22bba15001f\openssl-src-300.2.1+3.2.0\src\lib.rs:611:9:



  Error configuring OpenSSL build:
      Command: "perl" "./Configure" "--prefix=C:/msys64/home/ryuta/packages/rustpython/rustpython/target/release/build/openssl-sys-d6016873a007658f/out/openssl-build/install" "--openssldir=SYS$MANAGER:[OPENSSL]" "no-dso" "no-shared" "no-ssl3" "no-tests" "no-comp" "no-zlib" "no-zlib-dynamic" "--libdir=lib" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "no-capieng" "no-asm" "VC-WIN64A"
      Failed to execute: program not found```
Thank you for checking it!
I wish it is not a big problem for windows users. Windows does not have a system vendored openssl. If they could install openssl, installing openssl3 also will be possible.
Okay. I'm able to build on Windows 11. Here's how.

Use `winget` to install [Strawberry Perl](https://strawberryperl.com/)

```
winget install StrawberryPerl.StrawberryPerl
```

then

```
cargo build --release --features ssl-vendor
```

(It takes really long time to build `openssl-sys v0.9.98`. Please be patient!)

```
C:\msys64\home\ryuta\packages\rustpython>.\rustpython\target\release\rustpython.exe
Welcome to the magnificent Rust Python 0.3.1 interpreter 😱 🖖
RustPython 3.12.0
Type "help", "copyright", "credits" or "license" for more information.
>>>>> import sys
>>>>> sys.getwindowsversion()
sys.getwindowsversion(major=6, minor=2, build=9200, platform=2, service_pack='', service_pack_major=0, service_pack_minor=0, suite_mask=256, product_type=1, platform_version=(6, 2, 9200))
>>>>>
```

Unfortunately, `rustpython -m ensurepip` failed. ~~Can I file it as a separate issue?~~
Already reported as in https://github.com/RustPython/RustPython/issues/5146. It is related to missing `_ctypes`.
I successfly install the rustpython with `--features ssl-vendor`. but the method fail to install pip.
`SystemError: install-pip requires rustpython be build with '--features=ssl'`

My System info：
Linux: Ubuntu 20.04
Rust: rustc 1.76.0
Openssl: openssl (1.1.1f-1)



`rustpyhon -m ensurepip` works on AlmaLinux.
@carrt123 Thank you, #5330 will fix it.
@oroppas If you want to avoid perl, how about using ssl-vendor for linux and just ssl for windows? Our CI installs openssl with vcpkg.
@youknowone, does that mean the Windows users can `cargo build --release --features ssl` without explicitly `vcpkg install openssl`? That would definitely obviate the need to install perl and I love the idea!
@oroppas I am sorry, I am not good enough at windows ecosystem to say yes or no to the answer. Could you check if this command is working for your windows environment?

https://github.com/RustPython/RustPython/blob/main/.github/workflows/ci.yaml#L251-L256

The expected command is `cargo build --release --features ssl` as you said.
`cargo vcpkg build` seems to install openssl:

```
    Fetching vcpkg
    Checkout rev 2024.02.14
  Installing openssl
    Finished in 37.78s
```

but `cargo build --release --features ssl` fails with the same error:

```
error[E0599]: no method named `all_certificates` found for reference `&X509StoreRef` in the current scope
   --> stdlib\src\ssl.rs:706:18
    |
703 |               let certs = self
    |  _________________________-
704 | |                 .ctx()
705 | |                 .cert_store()
706 | |                 .all_certificates()
    | |                 -^^^^^^^^^^^^^^^^ method not found in `&X509StoreRef`
    | |_________________|
    |

For more information about this error, try `rustc --explain E0599`.
error: could not compile `rustpython-stdlib` (lib) due to 1 previous error
```
Isn't it because your windows env already have openssl 1.1.1 somewhere? openssl-sys has prioritized resolver. Probably vcpkg has lower priority than the other installation.
yeah, that's what I'm guessing. Let me do clean-up.
It still fails. I did `cargo build --verbose --release --features ssl`. The last line of the log caught my eyes:

```
 --cfg ossl101 --cfg ossl102 --cfg ossl110 --cfg ossl110g --cfg ossl111 --cfg "osslconf=\"OPENSSL_NO_SSL3_METHOD\""`
```
Does the mentioned PR fix openssl 1.x? Although its not in rustpython I use openssl 1.x forks over openssl 3 due to openssl not implementing QUIC properly.
@theoparis No, RustPython only supports openssl 3 now
There must be a patch we moved from 1.x to 3, but I have no idea which one. Trying revert may be worth if you need it. 
Okay. I needed to set `X86_64_PC_WINDOWS_MSVC_OPENSSL_DIR`, for example on my environment

PowerShell
```
PS D:\packages\rustpython\rustpython> cargo install cargo-vcpkg
PS D:\packages\rustpython\rustpython> cargo vcpkg build
PS D:\packages\rustpython\rustpython> $env:X86_64_PC_WINDOWS_MSVC_OPENSSL_DIR="D:\packages\rustpython\rustpython\target\vcpkg\packages\openssl_x64-windows-static-md"
PS D:\packages\rustpython\rustpython> cargo build --release --features ssl
```

Command Prompt
```
D:\packages\rustpython\rustpython> cargo install cargo-vcpkg
D:\packages\rustpython\rustpython> cargo vcpkg build
D:\packages\rustpython\rustpython> set X86_64_PC_WINDOWS_MSVC_OPENSSL_DIR=D:\packages\rustpython\rustpython\target\vcpkg\packages\openssl_x64-windows-static-md
D:\packages\rustpython\rustpython> cargo build --release --features ssl
```

`rustpython -m ensurepip` still fails with `ModuleNotFoundError: No module named '_ctypes'`
Hmm. `X86_64_PC_WINDOWS_MSVC_OPENSSL_DIR` may not be necessary after all.
Rebooting my windows machine and building from scratch without this environment variable still worked.
In any case, one less concern to build on windows!

Fix patch:
diff --git a/stdlib/build.rs b/stdlib/build.rs
index d8dad9b974..0109c633d7 100644
--- a/stdlib/build.rs
+++ b/stdlib/build.rs
@@ -1,29 +1,34 @@
 fn main() {
     println!(r#"cargo::rustc-check-cfg=cfg(osslconf, values("OPENSSL_NO_COMP"))"#);
-    println!("cargo::rustc-check-cfg=cfg(ossl101)");
-    println!("cargo::rustc-check-cfg=cfg(ossl102)");
-    println!("cargo::rustc-check-cfg=cfg(ossl110)");
-    println!("cargo::rustc-check-cfg=cfg(ossl110g)");
-    println!("cargo::rustc-check-cfg=cfg(ossl111)");
+
+    #[allow(clippy::unusual_byte_groupings)]
+    let ossl_vers = [
+        (0x1_00_01_00_0, "ossl101"),
+        (0x1_00_02_00_0, "ossl102"),
+        (0x1_01_00_00_0, "ossl110"),
+        (0x1_01_00_07_0, "ossl110g"),
+        (0x1_01_00_08_0, "ossl110h"),
+        (0x1_01_01_00_0, "ossl111"),
+        (0x1_01_01_04_0, "ossl111d"),
+        (0x3_00_00_00_0, "ossl300"),
+        (0x3_01_00_00_0, "ossl310"),
+        (0x3_02_00_00_0, "ossl320"),
+        (0x3_03_00_00_0, "ossl330"),
+    ];
+
+    for (_, cfg) in ossl_vers {
+        println!("cargo::rustc-check-cfg=cfg({cfg})");
+    }
+
     #[allow(clippy::unusual_byte_groupings)]
     if let Ok(v) = std::env::var("DEP_OPENSSL_VERSION_NUMBER") {
         println!("cargo:rustc-env=OPENSSL_API_VERSION={v}");
         // cfg setup from openssl crate's build script
         let version = u64::from_str_radix(&v, 16).unwrap();
-        if version >= 0x1_00_01_00_0 {
-            println!("cargo:rustc-cfg=ossl101");
-        }
-        if version >= 0x1_00_02_00_0 {
-            println!("cargo:rustc-cfg=ossl102");
-        }
-        if version >= 0x1_01_00_00_0 {
-            println!("cargo:rustc-cfg=ossl110");
-        }
-        if version >= 0x1_01_00_07_0 {
-            println!("cargo:rustc-cfg=ossl110g");
-        }
-        if version >= 0x1_01_01_00_0 {
-            println!("cargo:rustc-cfg=ossl111");
+        for (ver, cfg) in ossl_vers {
+            if version >= ver {
+                println!("cargo:rustc-cfg={cfg}");
+            }
         }
     }
     if let Ok(v) = std::env::var("DEP_OPENSSL_CONF") {
diff --git a/stdlib/src/ssl.rs b/stdlib/src/ssl.rs
index f4bd9b2ebb..dc64d18267 100644
--- a/stdlib/src/ssl.rs
+++ b/stdlib/src/ssl.rs
@@ -700,12 +700,14 @@ mod _ssl {
             vm: &VirtualMachine,
         ) -> PyResult<Vec<PyObjectRef>> {
             let binary_form = binary_form.unwrap_or(false);
-            let certs = self
-                .ctx()
-                .cert_store()
-                .all_certificates()
-                .iter()
-                .map(|cert| cert_to_py(vm, cert, binary_form))
+            let ctx = self.ctx();
+            #[cfg(ossl300)]
+            let certs = ctx.cert_store().all_certificates();
+            #[cfg(not(ossl300))]
+            let certs = ctx.cert_store().objects().iter().filter_map(|x| x.x509());
+            let certs = certs
+                .into_iter()
+                .map(|ref cert| cert_to_py(vm, cert, binary_form))
                 .collect::<Result<Vec<_>, _>>()?;
             Ok(certs)
         }


Test patch: