Code change:
Julia 1.5 is out, see: and we don't have it packaged. More over, see how defines our other Julias: Several issues: is pointing to a TLS version which is old IMO. Certainly there are breaking changes seen in , but since Nixpkgs is a somewhat rolling release distro, are we going to keep these older versions and default to them for ever? I've once tried to reach for the community to hear if there's a strict policy regarding such scenarios, but Anyway, I'd like to propose a change that will keep only the LTS version and the latest - currently 1.4, and perhaps make the attribute , point to the latest. Any thoughts from maintainers or users? cc
Also, now I noticed all of our Julias fail to build, e.g:
I agree. And the 1.5 release candidates are out, so this will become the latest version in the near future. Last time I looked, the package built the same deps for all versions, including 0.7. In versions 1.0 and later, there are actually fewer dependencies (e.g. arpack and fftw have been dropped), so the package should be cleaned up to not include these unnecessary libraries in the build. I also think the default version should be the latest version (i.e., unless somebody explicitly installs julia-1-lts, they should get the latest).
OK I'm glad I have your support. I'll let this issue stay open for a while and hopefully I'll be able to take care of this a month or so from now.
cc
Back when Julia tests were not so messed up, I think there was often 1 previous version. I guess with 1.3 needing overriding to no tests, it is fine to just have julia-1-lts and Julia 1.4 as julia once Julia 1.4 gets built fine with some tests.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
Haven't had a lot of time to work on julia in nixpkgs, so just posting this as a FYI. We used multiple versions of julia at work, but we use a custom derivation, similar to this: We also do not run the tests, but in general it has been working great. Previously we also used the binary builds patched with patchelf, as there were some differences in the structure of the results of the vanilla binary build provided by julia and what we got when building from source. We gave up on using 'system' libraries, and build the releases that julia provides, as julia is very specific in the versions for its dependencies (LLVM patches mostly).
I tried to use that tarball in order to write a derivation from scratch. Here's my attempt: It fails with: Naturally I didn't further attempt to add curl or wget to the inputs, as that would fail in a sandbox as well. I've opened an issue upstream: complaining that that tarball doesn't really contain all dependencies.
I ran with sandbox enabled, and the expression I linked with works for me. So I think it is fine to just add curl to the deps, as with the full tarball, it isn't actually used.
There is another approach to avoid running tests: using precompiled binary. I tried downloading the 1.4.2 binary from and then afterward ran on the binary, then I got a running Julia interpreter just like that.
We did the same for a while using code similar to this: Unfortunately, we actually sometimes need to patch Julia, so we switched to the julia expressions from source, that I posted earlier.
Thanks that expression works well thanks to . I'm testing it now.
OK so after a long build, I got a working Julia derivation with these references: And this closure size: Which I must say it is satisfying. The question is, is this how we'd want it to be built? Using all of their dependencies and practically non of ours? Some of their dependencies are required to be patched, according to and they also specify that using the OS' libraries is not officially supported? Anyway I'd happy to get help on deciding this.
does this lead to Julia tests passing? I think for every dependency they patch, it is OK to just use their patched version. I guess it could be nice to use a separate build unpatched (but using the exact same version sounds like a likely restriction), but whatever dependency does not work right away can be switched to bundled.
I think it did, I've set and the build passed but I haven't inspected the log much, but here it is: Currently produces this log but there's no PR yet as I think there are many makeFlags that should probably be set as well. If anyone feels confident to open a PR with this feel free to ping me for a review. I agree with the idea, but what's bothering me is that it's not absolutely clear what versions are patched and which are not - the docs vs what's actually found in is not totally inline. Also, it seems that some of the patches are old, and were suggested upstream (to the libraries) but they weren't merged there for some reason. I'm still investigating, See My current target is to decide what libraries to use our version vs their version, with the preference to use our version, in order to hopefully reduce closure size due to sharing shared libraries. The idea is to decide this by checking whether the libraries have included the patch Julia suggests, and then whether the library's maintainer has released a new version that includes Julia's maintainers' patch, and whether we ship that new version.
does this lead to Julia tests passing? I think this log is cut short before the interesting part…
I've uploaded it here:
Hmmm. Reading the log does not convince me the tests were run… Pity.
Since 1.5 was just released, the title should probably be updated to be 1.5.
Downside with the patching approach, is users lose the ability to compile against MKL, which has HUGE performance increase (last I saw showed that MKL was ~4x faster at matrix multiplication!!). Would need to also package JuliaPro to get MKL. Of course, anything working nothing working, although would be a shame to lose compilation options.
Says
Thanks garrison, I saw that, but I have a feeling it won't play nicely with Nix...although if we go patching route, perhaps could do that during buildPhase
Are you concerned about specifically or all packages that use BinaryBuilder? There are a lot of them. More on this at
As I understand binary builder, that all happens in userspace, but modifies , which is in /nix/store
The installation instructions for do not seem to require root. My guess is that it stores the modified sysimage somewhere in . Am I wrong?
I just tried it with Julia 1.4 installed with Nix, and as I suspected fails to build due to
We need to have a interface - that would support building binary packages, and make the interpreter aware of them. I wish I was a bit more motivated to do this, but upstream disappoints me regarding the humble task of compiling Julia as is, with the right selection of a few system dependencies - no response at all at & . I'm also dis-motivated to explore this aspect, since their build system is of plain makefiles, and they keep patches of dependencies from so long ago, without documenting whether they are needed / used or not...
Nix behind It is a shame to see julia's support in NixOS languish. It seems pretty clear to me that upstream is only interested in supporting the configuration obtained by cloning their repository and typing on a stock Linux distribution. I understand their perspective in many ways, as having julia and all of its packages "just work" requires a lot of care, and building with unsupported configurations is very likely to result in brokenness somewhere in the package ecosystem if not in julia itself. Julia is a bit special in this regard because of its emphasis on performance and also its need to support interfaces to other languages/frameworks that often have weird requirements. I have found that typically, there is very good reason for these patches, even though at times they may not be documented except in git and github history. (Admittedly, I am less involved in upstream than I used to be.)
I think Julia team initially evaluated Nix as their binary distributor () but decided against it—there was a discussion somewhere but can’t find it. IIRC lack of Windows / desire to tie into native pkg management systems were factors. Julia’s does a pretty great job, and uses a solver to properly resolve dependency versions based on constraints. I’ve been incredibly frustrated by the withPackages pattern, at least in python. I constantly run into problems due to nixpkgs ignoring version constraints. There are some other nice advantages to nix of course: binary dependencies are shared across language ecosystems for example, and there are lots of supremely annoying packages (I’m looking at you CUDA...) that Nix makes delightfully easy to install, use, and maintain parallel versions. But until Nixpkgs supports multiple versions across the board, and has a solver baked in, I’m honestly not sure Nix brings enough advantages to bail on Edit: I think the approach done by poetry2nix and Mach-nix are the way to go for Julia packages + nix. yarn2nix is another shining example. I would love to see a solution in this vein—didn’t mean to douse any enthusiasm for people interested in helping improve the Julia situation on NixOS :)
Maybe it is worth to reaching out to Julia Computing folks at to get their attention on ?
We'd probably need to put this information in our attribute sets, just like our Python packages know what deps they need. Hmm that's a bit surprising to hear as the Python ecosystem seems most well organized. Have you ever encountered a real bug with a mismatched version of a dependency with a specific version specified in a package's ? IIRC fails if a version is used and it's not exactly what's available. If you've ever felt there's a certain python package of a certain version missing, have you considered contributing it? Even if not, was it too hard to override it? What do you mean "across the board"? That might be much more harder then it sounds, due to the constraints of Nix it self when being evaluated everytime you run and friends, see It's not a question of whether Nix brings an advantage or not, it's a question of whether there are any Julia packages that are built at other distros imperatively with a , and this produces (or even worse, downloads) shared objects or executables into (not in ), linked to Julia or other Nix deps that do reside in . If the answer is yes, than every Nix user will experience breakage after Julia is updated in his - not necessarily if it's a version bump of Julia, it could be a mere rebuild of Julia by hydra. Yea Mach-nix is an awesome idea well implemented ( thanks for the inspiration :heart:). I've also reached a similar conclusion at . However, this is my opinion for Rust and Go, and I tend to think that for a small ecosystem like Julia, perhaps it'll be too much - it shouldn't be that hard to manage all of these pacakges like Python is managed in Nixpkgs.
I opened
Frequently. Especially with Scipy, matplotlib, bokeh, sklearn, tensorflow, pytorch, etc... Often, nixpkgs used to have it until it was updated. Except in very rare circumstances, we do not keep more than one version of a python package around. I have no interest wasting time working around bugs that authors tried to spare me by specifying precise versions. It seems a sort of cognitive dissonance in our next community that we all care so deeply about reproducible builds, but for convenience often disregard the officially supported dependencies. As a result, there are often subtle bugs in software on NixOS that do not appear on other platforms. (Eg x11vnc crashes, shared memory problems, etc). Nix supports crazy high specificity in versioning, well we past semantic versioning down the the bit in many cases. And yet in nixpkgs, by convention, we remove the old version anytime we update. It’s understandable that Hydra cannot build everything, but it’s always bothered me that I have to pin nixpkgs if I want a build to stay functional over 1+ years. is mostly declarative. Of course, Julia is constantly re-compiling code, and until there is true support for precompilation, I don’t see how it’s possible to use nix store. This is due to multiple dispatch. Suppose I wrote a library to do an efficient k-nearest-neighbor query using some fancy free structure. Perhaps the core algorithms simply require a function for your types. When you write this new function, Julia has to compile all the code in the k-nearest neighbor library for this new type. Thus, it is not possible right now to compile a package on its own as the core pattern in Julia involves extending the function of other packages via multiple dispatch. You can only compile a package in context of the code that is running. Therefore, the Nix store approach is not appropriate. See
I see. You are talking about "authors" of Nixpkgs' expressions? I'm still a bit perplexed, as I earlier said, and correct me if I'm wrong: should fail if is used in , and the exact version is not available. Sometimes (), we do hardcode such version requirement changes in our expressions, but hopefully that's done only when it's justified. I agree (maybe a bit ). Generally I think, every dep's version related issue should be handled by someone like you, who's familiar with python packages you use. I'd at least (open PRs) to fix the issues I encounter :). I see. I guess the situation is more complex than I thought, but I don't think I'd be able to form a strong opinion until I'll get a Julia interpreter from an expression I'm satisfied with. Thanks for the link :).
Unfortunately, this pattern is done constantly (including in my own contributions!). The problem is that, except an extremely rare cases like tensorflow, nixpkgs does not allow multiple versions of same python package to co-exist. Every time we update a package, we remove the previous version from nixpkgs. Confounding this issue, we often disable tests because sometimes the effort is just too high when developers bake in assumptions that aren’t compatible with the Nix ethos (for some reason this seems to happen more with tests then source code...). The most frequent cause of failure is the code needs to download something to run the test. (I’ve tried to lobby in the past that the test phase should be permitted to have Internet access as long as it doesn’t modify /nix/store, but I digress). In any case, the end result is that using a python package on Nix typically means you aren’t using the exact dependencies that upstream declared, and you’re more likely to have a package that despite having changes in dependencies has never had the test run. Thus, amazingly, I’ve had more luck with Julia packages working on NixOS than Python when all other binaries are present in environment.
I don't think there's a strong policy in our Python ecosystem against multiple versions of packages - if this is what upstream recommends. Anyway, the whole topic is out of scope for this issue. Feel free to open a new issue and cc me and who's working hard to ship a 20.09 release.
Yes, sorry getting a little off topic here. I mainly mention it to suggest that our efforts in Julia that start from scratch Can learn from these issues. I’m not too familiar with the new binary builder, but was quite nice in that it would first search for libraries before attempting to install. See this issue for some context: Note that a core Julia dev gave a pretty great endorsement on Nix (albeit back in 2014): I think it might be most productive to do this development outside of nixpkgs, and do it in Julia to hook in with nixpkgs as needed. That way users could seamlessly use Pkg in Julia. I’d love to and have it just work instead of the current hack that I wrote that a lot of people seem to be using ()
Short summary of the Discourse thread: for LLVM and libuv it is essential to use the patches/forks. For other packages it is probably unnecessary to use the exact version specified by upstream.
Thanks for being there with me :). I think I'm gonna work on this once we are through with , to avoid merge conflicts.
TIL Guix manages to package Julia 1.4.1: What's interesting, is that they but they build it with Julia's LLVM patches fetched from the internet - they don't rely on Julia's build system to build it's own LLVM. For similar dependencies which the real system's version can't be used, they do the same. I think we should try to copy this approach.
FWIW I did that in Fedora originally, but I stopped. Since LLVM makes breaking releases very quickly, you'll probably want to move to a new version that Julia doesn't support quite soon.
Maybe I don't understand, but why should I move to an LLVM version that julia doesn't support? Also, aren't the and versions don't introduce breaking changes between minor releases? We could stick to since that's what Guix does and I think Julia uses this version as a base too.
I don't really know how NixOS works, hence the "FWIW". If you used versioned LLVM packages and keep older major versions then it's probably OK.
Just offering a user perspective: Julia provides a package manager that provides mostly reproducible (in a practical/scientific reproducibility sense rather than a bit-identical sense) environments easily from manifests. Increasingly, these environments are isolated from the host OS by binary packages for dependencies being served by Julia. I don't want Nix to manage dependencies that can handle for me: Julia already handles those excellently and conveniently. I just want Nix to handle the (usually few) system dependencies. What I would like as a NixOS user is for Nix to provide me with a working Julia 1.5 derivation that provides: julia, a FHS wrapper or something providing compatible versions of the few things that Julia's binary packages need (just libc, libstdc++, libgfortran; I think. ). When I say "compatible", I mean that these libraries should work with the binaries and with the julia binary provided by Nix. Not all Julia packages are BinaryBuilder-ised yet, so I'd also want an easy way to specify that I want compatible versions of other packages installed (e.g. Qt, cairo, etc). For packages that need python, I don't know if I'd like Julia or Nix to handle installing and managing that. The simplest approach would be to tell Julia to do it (there is pretty good support in Julia for distributing isolated environments of python + packages + dependencies; better than in , imo). I don't know how doable any of that is in Nix, just describing what a good outcome would look like to me.
I think this is quite doable. I would propose that we add a new package, , that is a "batteries included" build that contains system libraries for all common Julia packages. Would this approach work for BinaryBuilder? I haven't used this yet. If not, we could fall back to a FHSuserenv, but I'm reluctant to do so if at all avoidable as there are outstanding nix bugs () that create seg faults in some heavy data processing workflows.
I think that is a good proposal and I think it will work with BinaryBuilder packages (e.g. SDL2_jll). In any case, some approach that works with BinaryBuilder packages should be found because those are a critical part of the Julia environment now :) Some important packages (e.g. GR) still want to build and install their own software dynamically, and that doesn't work well in Nix, but most of those can plausibly move to the BinaryBuilder system (PR for GR ), so I think letting those be broken would be tolerable (and a big improvement over the status quo of Julia on nix). Perhaps we can get something working quickly by just patchelf'ing the official Julia builds for now? (I don't understand how all of this stuff works!)
Is there anything presently blocking just getting julia 1.5 into nixpkgs? I don't care about ecosystem working with nixpkgs, just a good ol' fashioned julia install.
yes, see
If you just want Julia and don't mind about binary packages breaking you can just download Julia from the official site and patchelf it. On Sun, 4 Oct 2020, 09:32 Tyler Benster, <:
On the there was also the idea floated to patch to patchelf dynamically loaded binary dependencies.
Guix has a Julia package and it works quite well. Perhaps some ideas can be borrowed from there: It'd be really useful to get Julia working on Nix. It's the only big outstanding thing that doesn't work and prevents me from migrating some servers in my organization.
I tried to copy their expressions in and it doesn't work yet, but there's also another seemingly working attempt by at .
So, do we want anything else now that 1.5.3 is packaged (:greenheart: ? I think we: 't need clean up the expressions a bit - not use the file if it's not shared. I did (2) along with the julia_10 fix in As for (1) I will need a bit of feedback.
I guess some people like me still use julia_13 via test-disabling overrides, so a bit of time after 1.5 reaches channels for smooth transition could make sense?
Maybe backport 1.5 to 20.09?
OK so I disabled the tests for julia13 in and made the default there as well. All merged pull requests referenced here should be backported.

Related discussion:
text: enableParallelBuilding = true; <del> doCheck = !stdenv.isDarwin; </del> <ins> # Other versions of Julia pass the tests, but we are not sure why these fail. doCheck = false; </ins> checkTarget = "testall"; # Julia's tests require read/write access to $HOME preCheck = ''