Code change:
One of the easiest ways to make CI faster is to make things parallel and simply use the hardware we have available to us. Unfortunately though we don't have a lot of data about how parallel our build is. Are there steps we think are parallel but actually aren't? Are we pegged to one core for long durations when there's other work we could be doing? The general idea here is that we'd spin up a daemon at the very start of the build which would sample CPU utilization every so often. This daemon would then update a file that's either displayed or uploaded at the end of the build. Hopefully we could then use these logs to get a better view into how the builders are working during the build, diagnose non-parallel portions of the build, and implement fixes to use all the cpus we've got. cc
On Windows this can be done by taking advantage of job objects. If the entire build is wrapped in a job object then we can call with to get a bunch of useful data.
I made script that will print output into the travis log every 30 seconds , . Some findings: Cloning submodules jemalloc, libcompilerbuildtins and liblibc alone takes 30 seconds. While building bootstrap, compiling serdederive, serdejson and bootstrap crates seems to take 30 seconds (total build time: 47 seconds). stage0: Compiling tidy crate seems to take around 30 seconds. Compiling rustcerrors takes at least 2 minutes, only one codegen-unit is used Compiling syntaxext takes 9 minutes, only one CGU used stage0 codegen artifacts: Compiling rustcllvm takes 1,5 minutes, one CGU During stage1, rustcerrors and syntaxext builds are approximately as slow as during stage0, rustc_plugins 2 minutes, one CGU. stage2: rustdoc took 2 minutes to build, one CGU compiletest suite=run-make mode=run-make: It looks like there is a single test that takes around 3 minutes to complete and has no parallelization. Testing alloc stage1: building liballoc takes around a minute Testing syntax stage1: building syntax takes 1.5 minutes, one CGU Notes: When the load average dropped towards 1, I assumed only one codegen unit was active. The script was only applied to the default pullrequest travis-ci configuration.
As shown in , the CPUs assigned to each job may have some performance difference: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz The clock-rate 2.4 GHz vs 2.5 GHz shouldn't make any noticeable difference though (this would at most slow down by 7.2 minutes out of 3 hours if everything is CPU-bound). It is not enough to explain the timeout in .
I was working on recently for this where it periodically prints out the CPU usage as a percentage for the whole system (aka 1 core on a 4 core machine is 25%). I only got Linux/OSX working though and was unable to figure out a good way to do it on Windows. My thinking for how we'd do this is probably download a binary near the beginning of the build (or set up some script). We'd then run just before we run . That way we could correlate the two timestamps of each log (the main log and the to similar moments in time. Initially I was also thinking we'd just at the end of the build and scrape it later if need be.

Related discussion:
text: <ins> #!/usr/bin/env python # ignore-tidy-linelength # This is a small script that we use on CI to collect CPU usage statistics of # our builders. By seeing graphs of CPU usage over time we hope to correlate # that with possible improvements to Rust's own build system, ideally diagnosing # that either builders are always fully using their CPU resources or they're # idle for long stretches of time. # # This script is relatively simple, but it's platform specific. Each platform # (OSX/Windows/Linux) has a different way of calculating the current state of # CPU at a point in time. We then compare two captured states to determine the # percentage of time spent in one state versus another. The state capturing is # all platform-specific but the loop at the bottom is the cross platform part # that executes everywhere. # # # Viewing statistics # # All builders will upload their CPU statistics as CSV files to our S3 buckets. # These URLS look like: # #   https://$bucket.s3.amazonaws.com/rustc-builds/$commit/cpu-$builder.csv # # for example # #   https://rust-lang-ci2.s3.amazonaws.com/rustc-builds/68baada19cd5340f05f0db15a3e16d6671609bcc/cpu-x86_64-apple.csv # # Each CSV file has two columns. The first is the timestamp of the measurement # and the second column is the % of idle cpu time in that time slice. Ideally # the second column is always zero. # # Once you've downloaded a file there's various ways to plot it and visualize # it. For command line usage you can use a script like so: # #      set timefmt '%Y-%m-%dT%H:%M:%S' #      set xdata time #      set ylabel "Idle CPU %" #      set xlabel "Time" #      set datafile sep ',' #      set term png #      set output "printme.png" #      set grid #      builder = "i686-apple" #      plot "cpu-".builder.".csv" using 1:2 with lines title builder # # Executed as `gnuplot < ./foo.plot` it will generate a graph called # `printme.png` which you can then open up. If you know how to improve this # script or the viewing process that would be much appreciated :) (or even if # you know how to automate it!) import datetime import sys import time if sys.platform == 'linux2': class State: def __init__(self): with open('/proc/stat', 'r') as file: data = file.readline().split() if data[0] != 'cpu': raise Exception('did not start with "cpu"') self.user = int(data[1]) self.nice = int(data[2]) self.system = int(data[3]) self.idle = int(data[4]) self.iowait = int(data[5]) self.irq = int(data[6]) self.softirq = int(data[7]) self.steal = int(data[8]) self.guest = int(data[9]) self.guest_nice = int(data[10]) def idle_since(self, prev): user = self.user - prev.user nice = self.nice - prev.nice system = self.system - prev.system idle = self.idle - prev.idle iowait = self.iowait - prev.iowait irq = self.irq - prev.irq softirq = self.softirq - prev.softirq steal = self.steal - prev.steal guest = self.guest - prev.guest guest_nice = self.guest_nice - prev.guest_nice total = user + nice + system + idle + iowait + irq + softirq + steal + guest + guest_nice return float(idle) / float(total) * 100 elif sys.platform == 'win32': from ctypes.wintypes import DWORD from ctypes import Structure, windll, WinError, GetLastError, byref class FILETIME(Structure): _fields_ = [ ("dwLowDateTime", DWORD), ("dwHighDateTime", DWORD), ] class State: def __init__(self): idle, kernel, user = FILETIME(), FILETIME(), FILETIME() success = windll.kernel32.GetSystemTimes( byref(idle), byref(kernel), byref(user), ) assert success, WinError(GetLastError())[1] self.idle = (idle.dwHighDateTime << 32) | idle.dwLowDateTime self.kernel = (kernel.dwHighDateTime << 32) | kernel.dwLowDateTime self.user = (user.dwHighDateTime << 32) | user.dwLowDateTime def idle_since(self, prev): idle = self.idle - prev.idle user = self.user - prev.user kernel = self.kernel - prev.kernel return float(idle) / float(user + kernel) * 100 elif sys.platform == 'darwin': from ctypes import * libc = cdll.LoadLibrary('/usr/lib/libc.dylib') PROESSOR_CPU_LOAD_INFO = c_int(2) CPU_STATE_USER = 0 CPU_STATE_SYSTEM = 1 CPU_STATE_IDLE = 2 CPU_STATE_NICE = 3 c_int_p = POINTER(c_int) class State: def __init__(self): num_cpus_u = c_uint(0) cpu_info = c_int_p() cpu_info_cnt = c_int(0) err = libc.host_processor_info( libc.mach_host_self(), PROESSOR_CPU_LOAD_INFO, byref(num_cpus_u), byref(cpu_info), byref(cpu_info_cnt), ) assert err == 0 self.user = 0 self.system = 0 self.idle = 0 self.nice = 0 cur = 0 while cur < cpu_info_cnt.value: self.user += cpu_info[cur + CPU_STATE_USER] self.system += cpu_info[cur + CPU_STATE_SYSTEM] self.idle += cpu_info[cur + CPU_STATE_IDLE] self.nice += cpu_info[cur + CPU_STATE_NICE] cur += num_cpus_u.value def idle_since(self, prev): user = self.user - prev.user system = self.system - prev.system idle = self.idle - prev.idle nice = self.nice - prev.nice return float(idle) / float(user + system + idle + nice) * 100.0 else: print('unknown platform', sys.platform) sys.exit(1) cur_state = State(); print("Time,Idle") while True: time.sleep(1); next_state = State(); now = datetime.datetime.utcnow().isoformat() idle = next_state.idle_since(cur_state) print("%s,%s" % (now, idle)) sys.stdout.flush() cur_state = next_state </ins>