joyent/illumos-extra — pull request #3 by richlowe (2012-02-28 03:53:07+00:00)
Title: openssl: Build with GCC but in SUNW-fashion

As with SunC, when building with GCC but for SunOS we need to make sure
that the integer type used by RC4 is always `char`. This means defining
`RC4_CHAR` rather than `RC4_INT` but also disabling the use of the x86 ASM
RC4 implementation, which also assumes `RC4_INT` (this is what was missed
in 070825).

This is not exactly complete, in that someone should check the other differences between `...-cc` and `...-cc-sunw` and whether they should be applied (the ones that stand out are various linker flags for direct binding, and mapfiles we tend to use in the OS). And other differences between `...-cc-sunw` and `...-gcc` and whether they're _really_ gcc-isms, or whether they are deviations from what we need.

When re-applying the changes I decided to use a -gcc-sunw to go with -cc-sunw to make the above comparisons easier, though I myself have only glanced at them, but these include the definition of `RC4_INDEX` and `DES_RISC1`

I've tested this, as far as it goes, by preloading a fixed libcrypto for a debug sshd and connecting to it both in general, and non-interactively with a DSA key (which is what triggers the most recent breakage)

@JohnSonnenschein and @jclulow have all the rest of the backstory here.