Code change:
<ins> #!/usr/bin/env python # ignore-tidy-linelength # This is a small script that we use on CI to collect CPU usage statistics of # our builders. By seeing graphs of CPU usage over time we hope to correlate # that with possible improvements to Rust's own build system, ideally diagnosing # that either builders are always fully using their CPU resources or they're # idle for long stretches of time. # # This script is relatively simple, but it's platform specific. Each platform # (OSX/Windows/Linux) has a different way of calculating the current state of # CPU at a point in time. We then compare two captured states to determine the # percentage of time spent in one state versus another. The state capturing is # all platform-specific but the loop at the bottom is the cross platform part # that executes everywhere. # # # Viewing statistics # # All builders will upload their CPU statistics as CSV files to our S3 buckets. # These URLS look like: # #   https://$bucket.s3.amazonaws.com/rustc-builds/$commit/cpu-$builder.csv # # for example # #   https://rust-lang-ci2.s3.amazonaws.com/rustc-builds/68baada19cd5340f05f0db15a3e16d6671609bcc/cpu-x86_64-apple.csv # # Each CSV file has two columns. The first is the timestamp of the measurement # and the second column is the % of idle cpu time in that time slice. Ideally # the second column is always zero. # # Once you've downloaded a file there's various ways to plot it and visualize # it. For command line usage you can use a script like so: # #      set timefmt '%Y-%m-%dT%H:%M:%S' #      set xdata time #      set ylabel "Idle CPU %" #      set xlabel "Time" #      set datafile sep ',' #      set term png #      set output "printme.png" #      set grid #      builder = "i686-apple" #      plot "cpu-".builder.".csv" using 1:2 with lines title builder # # Executed as `gnuplot < ./foo.plot` it will generate a graph called # `printme.png` which you can then open up. If you know how to improve this # script or the viewing process that would be much appreciated :) (or even if # you know how to automate it!) import datetime import sys import time if sys.platform == 'linux2': class State: def __init__(self): with open('/proc/stat', 'r') as file: data = file.readline().split() if data[0] != 'cpu': raise Exception('did not start with "cpu"') self.user = int(data[1]) self.nice = int(data[2]) self.system = int(data[3]) self.idle = int(data[4]) self.iowait = int(data[5]) self.irq = int(data[6]) self.softirq = int(data[7]) self.steal = int(data[8]) self.guest = int(data[9]) self.guest_nice = int(data[10]) def idle_since(self, prev): user = self.user - prev.user nice = self.nice - prev.nice system = self.system - prev.system idle = self.idle - prev.idle iowait = self.iowait - prev.iowait irq = self.irq - prev.irq softirq = self.softirq - prev.softirq steal = self.steal - prev.steal guest = self.guest - prev.guest guest_nice = self.guest_nice - prev.guest_nice total = user + nice + system + idle + iowait + irq + softirq + steal + guest + guest_nice return float(idle) / float(total) * 100 elif sys.platform == 'win32': from ctypes.wintypes import DWORD from ctypes import Structure, windll, WinError, GetLastError, byref class FILETIME(Structure): _fields_ = [ ("dwLowDateTime", DWORD), ("dwHighDateTime", DWORD), ] class State: def __init__(self): idle, kernel, user = FILETIME(), FILETIME(), FILETIME() success = windll.kernel32.GetSystemTimes( byref(idle), byref(kernel), byref(user), ) assert success, WinError(GetLastError())[1] self.idle = (idle.dwHighDateTime << 32) | idle.dwLowDateTime self.kernel = (kernel.dwHighDateTime << 32) | kernel.dwLowDateTime self.user = (user.dwHighDateTime << 32) | user.dwLowDateTime def idle_since(self, prev): idle = self.idle - prev.idle user = self.user - prev.user kernel = self.kernel - prev.kernel return float(idle) / float(user + kernel) * 100 elif sys.platform == 'darwin': from ctypes import * libc = cdll.LoadLibrary('/usr/lib/libc.dylib') PROESSOR_CPU_LOAD_INFO = c_int(2) CPU_STATE_USER = 0 CPU_STATE_SYSTEM = 1 CPU_STATE_IDLE = 2 CPU_STATE_NICE = 3 c_int_p = POINTER(c_int) class State: def __init__(self): num_cpus_u = c_uint(0) cpu_info = c_int_p() cpu_info_cnt = c_int(0) err = libc.host_processor_info( libc.mach_host_self(), PROESSOR_CPU_LOAD_INFO, byref(num_cpus_u), byref(cpu_info), byref(cpu_info_cnt), ) assert err == 0 self.user = 0 self.system = 0 self.idle = 0 self.nice = 0 cur = 0 while cur < cpu_info_cnt.value: self.user += cpu_info[cur + CPU_STATE_USER] self.system += cpu_info[cur + CPU_STATE_SYSTEM] self.idle += cpu_info[cur + CPU_STATE_IDLE] self.nice += cpu_info[cur + CPU_STATE_NICE] cur += num_cpus_u.value def idle_since(self, prev): user = self.user - prev.user system = self.system - prev.system idle = self.idle - prev.idle nice = self.nice - prev.nice return float(idle) / float(user + system + idle + nice) * 100.0 else: print('unknown platform', sys.platform) sys.exit(1) cur_state = State(); print("Time,Idle") while True: time.sleep(1); next_state = State(); now = datetime.datetime.utcnow().isoformat() idle = next_state.idle_since(cur_state) print("%s,%s" % (now, idle)) sys.stdout.flush() cur_state = next_state </ins>

Related discussion:
text: The script was only applied to the default pullrequest travis-ci configuration.
As shown in , the CPUs assigned to each job may have some performance difference: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz The clock-rate 2.4 GHz vs 2.5 GHz shouldn't make any noticeable difference though (this would at most slow down by 7.2 minutes out of 3 hours if everything is CPU-bound). It is not enough to explain the timeout in .
I was working on recently for this where it periodically prints out the CPU usage as a percentage for the whole system (aka 1 core on a 4 core machine is 25%). I only got Linux/OSX working though and was unable to figure out a good way to do it on Windows. My thinking for how we'd do this is probably download a binary near the beginning of the build (or set up some script). We'd then run just before we run . That way we could correlate the two timestamps of each log (the main log and the to similar moments in time. Initially I was also thinking we'd just at the end of the build and scrape it later if need be.