Code change:
One of the easiest ways to make CI faster is to make things parallel and simply use the hardware we have available to us. Unfortunately though we don't have a lot of data about how parallel our build is. Are there steps we think are parallel but actually aren't? Are we pegged to one core for long durations when there's other work we could be doing? The general idea here is that we'd spin up a daemon at the very start of the build which would sample CPU utilization every so often. This daemon would then update a file that's either displayed or uploaded at the end of the build. Hopefully we could then use these logs to get a better view into how the builders are working during the build, diagnose non-parallel portions of the build, and implement fixes to use all the cpus we've got. cc
On Windows this can be done by taking advantage of job objects. If the entire build is wrapped in a job object then we can call with to get a bunch of useful data.
I made script that will print output into the travis log every 30 seconds , . Some findings: Cloning submodules jemalloc, libcompilerbuildtins and liblibc alone takes 30 seconds. While building bootstrap, compiling serdederive, serdejson and bootstrap crates seems to take 30 seconds (total build time: 47 seconds). stage0: Compiling tidy crate seems to take around 30 seconds. Compiling rustcerrors takes at least 2 minutes, only one codegen-unit is used Compiling syntaxext takes 9 minutes, only one CGU used stage0 codegen artifacts: Compiling rustcllvm takes 1,5 minutes, one CGU During stage1, rustcerrors and syntaxext builds are approximately as slow as during stage0, rustc_plugins 2 minutes, one CGU. stage2: rustdoc took 2 minutes to build, one CGU compiletest suite=run-make mode=run-make: It looks like there is a single test that takes around 3 minutes to complete and has no parallelization. Testing alloc stage1: building liballoc takes around a minute Testing syntax stage1: building syntax takes 1.5 minutes, one CGU Notes: When the load average dropped towards 1, I assumed only one codegen unit was active. The script was only applied to the default pullrequest travis-ci configuration.
As shown in , the CPUs assigned to each job may have some performance difference: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz The clock-rate 2.4 GHz vs 2.5 GHz shouldn't make any noticeable difference though (this would at most slow down by 7.2 minutes out of 3 hours if everything is CPU-bound). It is not enough to explain the timeout in .
I was working on recently for this where it periodically prints out the CPU usage as a percentage for the whole system (aka 1 core on a 4 core machine is 25%). I only got Linux/OSX working though and was unable to figure out a good way to do it on Windows. My thinking for how we'd do this is probably download a binary near the beginning of the build (or set up some script). We'd then run just before we run . That way we could correlate the two timestamps of each log (the main log and the to similar moments in time. Initially I was also thinking we'd just at the end of the build and scrape it later if need be.

Related discussion:
text: - checkout: self fetchDepth: 2 <ins> # Spawn a background process to collect CPU usage statistics which we'll upload # at the end of the build. See the comments in the script here for more # information. - bash: python src/ci/cpu-usage-over-time.py &> cpu-usage.csv & displayName: "Collect CPU-usage statistics in the background" </ins> - bash: printenv | sort displayName: Show environment variables