page_title: Supercomputer operating systems
section_title: Modern approaches
hierarchical_section_title: Supercomputer operating systems / Modern approaches
caption_attribution_description: English: The IBM Blue Gene/P supercomputer installation at the Argonne Leadership Angela Yang Computing Facility located in the Argonne National Laboratory, in Lemont, Illinois, USA.
context_page_description: Since the end of the 20th century, supercomputer operating systems have undergone major transformations, as fundamental changes have occurred in supercomputer architecture. While early operating systems were custom tailored to each supercomputer to gain speed, the trend has been moving away from in-house operating systems and toward some form of Linux, with it running all the supercomputers on the TOP500 list in November 2017.
  Given that modern massively parallel supercomputers typically separate computations from other services by using multiple types of nodes, they usually run different operating systems on different nodes, e.g., using a small and efficient lightweight kernel such as Compute Node Kernel or Compute Node Linux on compute nodes, but a larger system such as a Linux-derivative on server and input/output nodes.
context_section_description: The IBM Blue Gene supercomputer uses the CNK operating system on the compute nodes, but uses a modified Linux-based kernel called I/O Node Kernel (INK) on the I/O nodes. CNK is a lightweight kernel that runs on each node and supports a single application running for a single user on that node. For the sake of efficient operation, the design of CNK was kept simple and minimal, with physical memory being statically mapped and the CNK neither needing nor providing scheduling or context switching. CNK does not even implement file I/O on the compute node, but delegates that to dedicated I/O nodes. However, given that on the Blue Gene multiple compute nodes share a single I/O node, the I/O node operating system does require multi-tasking, hence the selection of the Linux-based operating system.
  While in traditional multi-user computer systems and early supercomputers, job scheduling was in effect a task scheduling problem for processing and peripheral resources, in a massively parallel system, the job management system needs to manage the allocation of both computational and communication resources. It is essential to tune task scheduling, and the operating system, in different configurations of a supercomputer. A typical parallel job scheduler has a master scheduler which instructs some number of slave schedulers to launch, monitor, and control parallel jobs, and periodically receives reports from them about the status of job progress.
  Some, but not all supercomputer schedulers attempt to maintain locality of job execution. The PBS Pro scheduler used on the Cray XT3 and Cray XT4 systems does not attempt to optimize locality on its three-dimensional torus interconnect, but simply uses the first available processor. On the other hand, IBM's scheduler on the Blue Gene supercomputers aims to exploit locality and minimize network contention by assigning tasks from the same application to one or more midplanes of an 8x8x8 node group. The Slurm Workload Manager scheduler uses a best fit algorithm, and performs Hilbert curve scheduling to optimize locality of task assignments. Several modern supercomputers such as the Tianhe-2 use Slurm, which arbitrates contention for resources across the system. Slurm is open source, Linux-based, very scalable, and can manage thousands of nodes in a computer cluster with a sustained throughput of over 100,000 jobs per hour.
caption: The Blue Gene/P supercomputer at Argonne National Lab
exif: {"Image ImageWidth": "2100", "Image ImageLength": "1391", "Image BitsPerSample": "[8, 8, 8, 8]", "Image Compression": "Uncompressed", "Image PhotometricInterpretation": "5", "Image Make": "NIKON CORPORATION", "Image Model": "NIKON D2H", "Image Orientation": "Horizontal (normal)", "Image SamplesPerPixel": "4", "Image XResolution": "300", "Image YResolution": "300", "Image PlanarConfiguration": "1", "Image ResolutionUnit": "Pixels/Inch", "Image Software": "Adobe Photoshop CS3 Macintosh", "Image DateTime": "2008:05:20 20:04:02", "Image ExifOffset": "308", "Thumbnail Compression": "JPEG (old-style)", "Thumbnail XResolution": "72", "Thumbnail YResolution": "72", "Thumbnail ResolutionUnit": "Pixels/Inch", "Thumbnail JPEGInterchangeFormat": "1038", "Thumbnail JPEGInterchangeFormatLength": "5729", "EXIF ExposureTime": "1", "EXIF FNumber": "14", "EXIF ExposureProgram": "Manual", "EXIF ISOSpeedRatings": "250", "EXIF ExifVersion": "0221", "EXIF DateTimeOriginal": "2007:12:10 16:36:55", "EXIF DateTimeDigitized": "2007:12:10 16:36:55", "EXIF ComponentsConfiguration": "YCbCr", "EXIF CompressedBitsPerPixel": "4", "EXIF ExposureBiasValue": "0", "EXIF MaxApertureValue": "3", "EXIF MeteringMode": "Pattern", "EXIF LightSource": "Unknown", "EXIF Flash": "Flash did not fire", "EXIF FocalLength": "14", "EXIF SubSecTime": "65", "EXIF SubSecTimeOriginal": "65", "EXIF SubSecTimeDigitized": "65", "EXIF FlashPixVersion": "0100", "EXIF ColorSpace": "sRGB", "EXIF ExifImageWidth": "2100", "EXIF ExifImageLength": "1391", "EXIF RelatedSoundFile": "", "EXIF SensingMethod": "One-chip color area", "EXIF FileSource": "Digital Camera", "EXIF SceneType": "Directly Photographed", "EXIF CVAPattern": "[0, 2, 0, 2, 1, 2, 0, 1]", "EXIF CustomRendered": "Normal", "EXIF ExposureMode": "Manual Exposure", "EXIF WhiteBalance": "Auto", "EXIF DigitalZoomRatio": "1", "EXIF FocalLengthIn35mmFilm": "21", "EXIF SceneCaptureType": "Standard", "EXIF GainControl": "None", "EXIF Contrast": "Normal", "EXIF Saturation": "Normal", "EXIF Sharpness": "Normal", "EXIF SubjectDistanceRange": "0"}