Code change:
<ins> #!/usr/bin/env python # ignore-tidy-linelength # This is a small script that we use on CI to collect CPU usage statistics of # our builders. By seeing graphs of CPU usage over time we hope to correlate # that with possible improvements to Rust's own build system, ideally diagnosing # that either builders are always fully using their CPU resources or they're # idle for long stretches of time. # # This script is relatively simple, but it's platform specific. Each platform # (OSX/Windows/Linux) has a different way of calculating the current state of # CPU at a point in time. We then compare two captured states to determine the # percentage of time spent in one state versus another. The state capturing is # all platform-specific but the loop at the bottom is the cross platform part # that executes everywhere. # # # Viewing statistics # # All builders will upload their CPU statistics as CSV files to our S3 buckets. # These URLS look like: # #   https://$bucket.s3.amazonaws.com/rustc-builds/$commit/cpu-$builder.csv # # for example # #   https://rust-lang-ci2.s3.amazonaws.com/rustc-builds/68baada19cd5340f05f0db15a3e16d6671609bcc/cpu-x86_64-apple.csv # # Each CSV file has two columns. The first is the timestamp of the measurement # and the second column is the % of idle cpu time in that time slice. Ideally # the second column is always zero. # # Once you've downloaded a file there's various ways to plot it and visualize # it. For command line usage you can use a script like so: # #      set timefmt '%Y-%m-%dT%H:%M:%S' #      set xdata time #      set ylabel "Idle CPU %" #      set xlabel "Time" #      set datafile sep ',' #      set term png #      set output "printme.png" #      set grid #      builder = "i686-apple" #      plot "cpu-".builder.".csv" using 1:2 with lines title builder # # Executed as `gnuplot < ./foo.plot` it will generate a graph called # `printme.png` which you can then open up. If you know how to improve this # script or the viewing process that would be much appreciated :) (or even if # you know how to automate it!) import datetime import sys import time if sys.platform == 'linux2': class State: def __init__(self): with open('/proc/stat', 'r') as file: data = file.readline().split() if data[0] != 'cpu': raise Exception('did not start with "cpu"') self.user = int(data[1]) self.nice = int(data[2]) self.system = int(data[3]) self.idle = int(data[4]) self.iowait = int(data[5]) self.irq = int(data[6]) self.softirq = int(data[7]) self.steal = int(data[8]) self.guest = int(data[9]) self.guest_nice = int(data[10]) def idle_since(self, prev): user = self.user - prev.user nice = self.nice - prev.nice system = self.system - prev.system idle = self.idle - prev.idle iowait = self.iowait - prev.iowait irq = self.irq - prev.irq softirq = self.softirq - prev.softirq steal = self.steal - prev.steal guest = self.guest - prev.guest guest_nice = self.guest_nice - prev.guest_nice total = user + nice + system + idle + iowait + irq + softirq + steal + guest + guest_nice return float(idle) / float(total) * 100 elif sys.platform == 'win32': from ctypes.wintypes import DWORD from ctypes import Structure, windll, WinError, GetLastError, byref class FILETIME(Structure): _fields_ = [ ("dwLowDateTime", DWORD), ("dwHighDateTime", DWORD), ] class State: def __init__(self): idle, kernel, user = FILETIME(), FILETIME(), FILETIME() success = windll.kernel32.GetSystemTimes( byref(idle), byref(kernel), byref(user), ) assert success, WinError(GetLastError())[1] self.idle = (idle.dwHighDateTime << 32) | idle.dwLowDateTime self.kernel = (kernel.dwHighDateTime << 32) | kernel.dwLowDateTime self.user = (user.dwHighDateTime << 32) | user.dwLowDateTime def idle_since(self, prev): idle = self.idle - prev.idle user = self.user - prev.user kernel = self.kernel - prev.kernel return float(idle) / float(user + kernel) * 100 elif sys.platform == 'darwin': from ctypes import * libc = cdll.LoadLibrary('/usr/lib/libc.dylib') PROESSOR_CPU_LOAD_INFO = c_int(2) CPU_STATE_USER = 0 CPU_STATE_SYSTEM = 1 CPU_STATE_IDLE = 2 CPU_STATE_NICE = 3 c_int_p = POINTER(c_int) class State: def __init__(self): num_cpus_u = c_uint(0) cpu_info = c_int_p() cpu_info_cnt = c_int(0) err = libc.host_processor_info( libc.mach_host_self(), PROESSOR_CPU_LOAD_INFO, byref(num_cpus_u), byref(cpu_info), byref(cpu_info_cnt), ) assert err == 0 self.user = 0 self.system = 0 self.idle = 0 self.nice = 0 cur = 0 while cur < cpu_info_cnt.value: self.user += cpu_info[cur + CPU_STATE_USER] self.system += cpu_info[cur + CPU_STATE_SYSTEM] self.idle += cpu_info[cur + CPU_STATE_IDLE] self.nice += cpu_info[cur + CPU_STATE_NICE] cur += num_cpus_u.value def idle_since(self, prev): user = self.user - prev.user system = self.system - prev.system idle = self.idle - prev.idle nice = self.nice - prev.nice return float(idle) / float(user + system + idle + nice) * 100.0 else: print('unknown platform', sys.platform) sys.exit(1) cur_state = State(); print("Time,Idle") while True: time.sleep(1); next_state = State(); now = datetime.datetime.utcnow().isoformat() idle = next_state.idle_since(cur_state) print("%s,%s" % (now, idle)) sys.stdout.flush() cur_state = next_state </ins>

Related discussion:
text: One of the easiest ways to make CI faster is to make things parallel and simply use the hardware we have available to us. Unfortunately though we don't have a lot of data about how parallel our build is. Are there steps we think are parallel but actually aren't? Are we pegged to one core for long durations when there's other work we could be doing? The general idea here is that we'd spin up a daemon at the very start of the build which would sample CPU utilization every so often. This daemon would then update a file that's either displayed or uploaded at the end of the build. Hopefully we could then use these logs to get a better view into how the builders are working during the build, diagnose non-parallel portions of the build, and implement fixes to use all the cpus we've got. cc
On Windows this can be done by taking advantage of job objects. If the entire build is wrapped in a job object then we can call with to get a bunch of useful data.
I made script that will print output into the travis log every 30 seconds , . Some findings: Cloning submodules jemalloc, libcompilerbuildtins and liblibc alone takes 30 seconds. While building bootstrap, compiling serdederive, serdejson and bootstrap crates seems to take 30 seconds (total build time: 47 seconds). stage0: Compiling tidy crate seems to take around 30 seconds. Compiling rustcerrors takes at least 2 minutes, only one codegen-unit is used Compiling syntaxext takes 9 minutes, only one CGU used stage0 codegen artifacts: Compiling rustcllvm takes 1,5 minutes, one CGU During stage1, rustcerrors and syntaxext builds are approximately as slow as during stage0, rustc_plugins 2 minutes, one CGU. stage2: rustdoc took 2 minutes to build, one CGU compiletest suite=run-make mode=run-make: It looks like there is a single test that takes around 3 minutes to complete and has no parallelization. Testing alloc stage1: building liballoc takes around a minute Testing syntax stage1: building syntax takes 1.5 minutes, one CGU Notes: When the load average dropped towards 1, I assumed only one codegen unit was active.