Compute Projecとしok Server rboard V3.1 Intel Mo Author: Jia Ning, Hardware Engineer, Facebook
11
Table 1

© 2014 Facebook. As of Oct 24, 2014, the following persons or entities have made this Specification available under the Open Web Foundation Final Specification Agreement (OWFa 1.0), which is available at: http://www.openwebfoundation.org/legal/the-owf-1-0-agreements/owfa-1-0: Facebook, Inc. You can review the signed copies of the Open Web Foundation Agreement Version 1.0 for this Specification at http://opencompute.org/licensing/, which may also include additional parties to those listed above. Your use of this Specification may be subject to other third party rights. THIS SPECIFICATION IS PROVIDED "AS IS." The contributors expressly disclaim any warranties (express, implied, or otherwise), including implied warranties of merchantability, noninfringement, fitness for a particular purpose, or title, related to the Specification. The entire risk as to implementing or otherwise using the Specification is assumed by the Specification implementer and user. IN NO EVENT WILL ANY PARTY BE LIABLE TO ANY OTHER PARTY FOR LOST PROFITS OR ANY FORM OF INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY CHARACTER FROM ANY CAUSES OF ACTION OF ANY KIND WITH RESPECT TO THIS SPECIFICATION OR ITS GOVERNING AGREEMENT, WHETHER BASED ON BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, AND WHETHER OR NOT THE OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. CONTRIBUTORS AND LICENSORS OF THIS SPECIFICATION MAY HAVE MENTIONED CERTAIN TECHNOLOGIES THAT ARE MERELY REFERENCED WITHIN THIS SPECIFICATION AND NOT LICENSED UNDER THE OWF CLA OR OWFa. THE FOLLOWING IS A LIST OF MERELY REFERENCED TECHNOLOGY: INTELLIGENT PLATFORM MANAGEMENT INTERFACE (IPMI), I²C TRADEMARK OF PHILLIPS SEMICONDUCTOR. IMPLEMENTATION OF THESE TECHNOLOGIES MAY BE SUBJECT TO THEIR OWN LEGAL TERMS.
2
Scope
This specification describes Intel Motherboard v3.0 design and ddesign requirement to integrate Intel Motherboard v3.0 into Open Rack V1² and Open Rack V2².
3

1 http://opencompute.org/wp/wp-content/uploads/2012/09/Open_Compute_Project_Open_Rack_v1.0.pdf 2 http://files.opencompute.org/oc/public.php?service=files&t=348f3df2cc4ce573397fcc4424f68ca6&download 4
7
Overview
4
Overview
4.1
Intel Motherboard V3.1 (also referred to "motherboard" or "the motherboard" in this document, unless noted otherwise) is based on Intel® Xeon® Processor E5-2600 v3 (formerly code-named Haswell-EP processor) product family CPU architecture. The motherboard supports up to 16 DIMMs. The motherboard is designed to work with Open Rack, including Open Rack V1 (ORv1) and Open Rack V2 (ORv2) described in the following chapter. Open Rack V1 is OCP Rack design in 2012, which integrates power supply in the rack. Intel Motherboard V3.1-ORv1 sled is the compute sled compatible with Open Rack V1. It can also serve as head node of storage box. Open Rack V2 is OCP Rack design in 2013. Intel Motherboard V3.1-ORv2 sled is the compute sled compatible with Open rack V2. It can also serve as head node of storage box.
4.2
Open Rack Introduction
This chapter gives background of Open Rack V1 and V2. The details of the motherboard's electrical and mechanical interfaces to Open Rack V1 and V2 are described in Chapter 12 and Chapter 13.
4.2.1
Open Rack V1 Introduction
Open Rack V1 has 3x power zones. Each power zone has 10x OpenU (OU)3 for IT equipment (server, storage, etc.), and 3x OU for power shelf. Each ORv1 power shelf has 6+1x 700W redundant PSU to provide 4200W continuous max loading and 3x bus bars for the power zone it serves. For each 10 OU server space, its height is 4δomm as front view shown in Figure 4-1.
Open Rack V2 Introduction
4.2.2
Open Rack V2 has 2x power zones. Each power zone has 16x OU for IT equipment (server, storage, etc.), and 3x OU for power shelf. Each ORv2 power shelf has 2+1x 3.3KW PSUs, and 3x Battery Backup Units (BBU) and provides 6.3KW4 continuous max loading through 1x bus bar to the power zone it is attached to.
Physical Specifications
5
5.1   Block Diagram
Figure 5-1 illustrates the functional block diagram of the Intel Motherboard V3.1. The 
dashed lines are for reserved connection, dual layout, and high-speed mid-plane optio
Placement and Form Factor
5.2
Board form factor is 6.5 inch by 20 inch (6.5"x20"). And Figure 5-2 illustrates board placement. The placement is meant to show key components’ relative positions, while exact dimension and position information would be exchanged by DXF format for layout and 3D model for mechanical. The vendor is responsible to do complete component placement. Form factor, CPU and DIMM socket location, PCIe slot position, front I/O ports’ position, Power and Reset buttons, PCIe Mezzanine Card connector position, mid- plane connectors, battery connector, and mounting holes should be followed strictly, while other components can be shifted based on layout routing as long as relative position is maintained. Following internal connectors should be placed as close as possible to front of the board in order to have easy front access: 1x vertical SATA signal connector . . 1x SATA power connector . 1x Debug card header . 1x mSATA connector (dual layout with one M.2 connector) . 1x RJ45 Following mid-plane connector footprints should be placed at backside of board to provide mid-plane connection co-layout: 1x AirMax VS2® 3x8 . . ıx AirMax® Guide . 2x AirMax® VS Power 2x2 The major changes of board placement compare to OCP Intel Motherboard Hardware V2.0 specs: Mounting Hole and thumb screw location is changed . . . Mid-plane connector interface is changed CPU and DIMM location is changed . . Remove 1x SATA and 1x miniSAS vertical connector Add mSATA connector and M.2 connector . . Change x16 PCIe slot to x24 PCIe slot Modify motherboard outline to accommodate mezzanine card with dual QSFP connector Figure 5-2 Intel Motherboard V3.1 Placement
CPU and Memory
5.3
² http://www.opencompute.org/wp/wp- content/uploads/2012/05/Open_Compute_Project_Intel_Motherboard_v2.0.pdf
5.3.1
CPU
. The motherboard uses Intel® Xeon® E5-2600 v3 (LGA2011-3) Product Family processors with TDP up to 145W. The features listed below must be supported by the motherboard Support two Intel® Xeon® E5-2600 v3 (LGA2011-3) Product Family processors up to 145W TDP, and vendors should engage with Intel to ensure the design ready for future processors . . Two full-width Intel QPI links up to 9.6 GT/s/direction Up to 18 cores per CPU (up to 36 threads with Hyper-Threading Technology). Up to 45MB last level cache . Single Processor mode is supported
5.3.2
DIMM
. The motherboard has DIMM subsystem designed as below: DDR4 direct attach memory support on CPUo and CPU1. . 4x channels DDR4 registered memory interface on each CPU 2x DDR4 slots per channel (total 16 DIMM) . . . Support RDIMM, LRDIMM Support SR, DR and QR DIMM . Support DDR4 speeds of 1600/1866/2133 . Up to maximum 1024 GB with 64GB DIMMs . Follow updated JEDEC DDR4 specification with 288 pin DIMM socket Memory support matrix for DDR4 is as Table 5-1
Board design shall allow Intel® Xeon® Processor to run at maximum POR memory speeds.
5.3.3
Non-Volatile DIMM (NVDIMM)
Besides traditional DDR4 DIMM, the motherboard needs to support NVDIMM on all DIMM slots. A power failure detection circuit need to be implemented to initiate 3x actions related to data transferring: 1) CPU cache flush; 2) memory controller write pending queue flush with ADR mechanism and 3) Issue SAVE# signal to NVDIMM pin 230 to move DRAM data to NAND. Due to system energy storing and timing requirement limitation, the logic of action 1) is disabled by default with resistor option to enable. Logic of action 2) and 3) is enabled by default with resistor option to disable. The ODM will work with a NVDIMM to implement the BIOS design. Under-Voltage based power failure detection circuit should also trigger separate CPU/DIMM/FAN throttling with separate resistor options for enable and disable. Default is enable to reduce power consumption and slow down voltage decaying on P12V. . . . . . . . Power off, and reboot triggered by OS DC cycle from Host Partition Reset (Write oxE to oxCF9)
5.4
PCH
NVDIMM should cover test cases below for data protection: Surprising AC off by removing node from bus bar Issuing raw write-read command to BMC to power cycle Hot-Swap-Controller DC off triggered by 4 seconds override from either front panel switch or BMC DC cycle from BMC (chassis power cycle command) Warm-reboot triggered by front panel switch or BMC . . . The motherboard uses Intel® C610 series chipset, which supports following features: 3x USB 3.0/2.0 ports: one for front connector; one for optional vertical onboard connector; one for BMC in-band firmware update. 1x mSATA connector from SATA port 4 co-layout with M.2 connector; 1x individual SATA 6Gps ports from SATA port 5; 1x miniSAS port from SATA port 0/1/2/3, 1x miniSAS port from sSATA0/1/2/3 1x PCIe x4 ports to M.2 connector SPI interface, connect to BMC to enable BMC the capability to perform BIOS upgrade and recovery . . . SMBUS interface (master & slave) Intel® Server Platform Services (SPS) 3.0 Firmware with Intel® Node Manager PECI access to CPU o SMLinko connect to BMC o o o o o Intel® Manageability Engine (ME) obtain HSC PMBus related information through SMLinko to BMC interface by default; BMC has direct connection to HSC PMBus to have flexibility of HSC PMBus related feature support Intel® ME SMLink1 connect to Hotswap controller PMBus interface with resistor option, default is disconnect Power capping capability Temperature sensors reading from BMC
PCle Usage
5.5
Intel® Xeon Processor E5-2600 v3 Product Family provides 40x PCIe Gen3 lanes and Intel® C610 provides 8x PCIe Gen2 lanes. PCIe lanes are configured according to Table 5-2. All PCIe lanes are connected to CPUo and PCH to support 1x CPU configuration. Intel® Xeon Processor E5-2600 v3 Product Family only supports NTB (non-transparent bridge) feature on PCIe port3a/IOU1 in 1x4, 1x8 or 1x16 configurations.

PCB Stack Up
5.6
Following PCB stack up should be followed for motherboard design. The vendor needs to check with PCB fab vendors to fine tune the impedance based on the impedance control table below before starting PCB design. Table 5-3 Motherboard PCB Stack Up

BIOS
6
Vendors shall be responsible for supplying and customizing the BIOS for the motherboard. The specific BIOS requirements are outlined in this section. Vendors must make changes to the BIOS at any point in the motherboard’s life cycle (development, production, and sustaining) upon request.
6.1
BIOS Chip
The BIOS chip should use PCH’s SPI interface through BMC controlled MUX for BMC to perform offline BIOS update or recovery. The vendor is responsible for selecting a specific BIOS chip, which should fit the required functionality in addition to potential additional features that may be required in the future. 16MByte size is recommended considering space needed for both BIOS and Intel® ME firmware. Vendor should provide flash region plan for different code and current used size for each region to justify the sizing of the SPI flash. A socket on the motherboard should be used to hold BIOS chip, so BIOS chip can be manually replaced. The BIOS socket must be easily accessible; other components on the motherboard must not interfere with the insertion or removal of the BIOS chip; The BIOS socket needs to fit JEDEC specification package considering tolerance, and fit major available SPI flash vendors' package drawing. A secondary identical BIOS chip is designed in sharing the same SPI bus with multiplexed CS pin, MUX is controlled by BMC.  More detail of this scheme is described in section 8.14.
6.2
BIOS Source Code
BIOS should be a UEFI system firmware. The vendor shall be responsible to maintain BIOS source code to make sure it has latest code release from Intel and UEFI system firmware code base vendors. Vendor shall provide an updated version tracker with each release.
6.3
BIOS Feature Requirements
6.3.1
Optimization
. The BIOS should be tuned to minimize system power consumption and maximize performance. This includes: Disable any unused devices, such as unused PCI, PCIe ports, USB ports, SATA/SAS ports, clock generator and buffer ports. Tuning CPU/Chipset settings to reach minimized power consumption and best performance in a data center environment. . . . Open Turbo Mode tuning option for PL1, PL2, PL1 clamp bit, PL2 clamp bit, short and long time duration. SPEC power should be used as guidance by ODM to validate BIOS tuning results.
6.3.2
Setup Menu
. The vendor should provide a BIOS specification, which includes a complete BIOS, setup menu and default settings. Requirements include but are not limited to: Settings for adjusting memory speed, QPI speed, Speed-step/Turbo mode and CPU Cx power state: The default follows the CPU and chipset vendor’s POR unless otherwise mentioned. . Settings to enable different Turbo mode tuning settings based on CPU SKU and memory configuration: The default is Turbo enabled with the CPU vendor’s POR, unless otherwise mentioned. Setting for the power feature after AC failure: The default is set to restore last power state. Setting for the local physical COM port (COMo) and SOL (COM1): The default is enable console redirection on both ports with baud rate 57600, no flow control, terminal type VT100, 8 data bits, No Parity, 1 Stop Bit. Setting for legacy console redirection to be local physical COM port (COMo) and SOL(COM1). The default is SOL(COM1) Setting for the altitude of the server deployment location: The default is 300M. Setting for the watchdog timer: The default setting for EVT/DVT/PVT is disabled. The default setting for MP is enabled. The timeout value is 15 minutes and reset the system after the timer expires. The watchdog timer is always disabled after POST. Setting for ECC error threshold: Available settings are 1, 4, 10 and 1000. The default setting is 4 for EVT, DVT, and PVT and 1000 for MP. Setting for ECC error event log threshold: Available settings are disabled, 10, 50, 100. The default setting is 10. . . . . . . . . . If a CMOS CRC error happens, the BIOS should load the system default automatically and log the CMOS clear event in SEL. The default setting to disable all “wait for keyboard input to continue” types of features is “not to wait for keyboard input to continue”. Calculate checksum of BIOS setting, display in BIOS setup menu, and output to SMBIOS table. Setting to save and load 10 different sets of user default. Setting of UEFI and Legacy boot options: The default is UEFI. Display SKU and hardware revision in main page based on BOARD ID and FAB ID. Setting of PPIN (Protected Processor Inventory Number) Control: The default setting is unlock/enable. Display RC version in main page. Display CPU information in main page including CPU signature, processor cores, and microcode patch version. Display memory information in main page including current memory speed, total memory capacity and type (DDR4). Display PCH information in main page including name and stepping. . . . . . . . . . .
Setting of Setup Prompt timeout: The default is “7 seconds”.
6.3.3
Boot Options
The BIOS must support PXE Boot capability in both IPv4 and IPv6 environment at the same time, and boot from SATA/SAS and USB interface. BIOS should provide boot option selection capability. The default boot device priority is: 1st: USB device if available . . 2nd: Mezzanine card NIC IPv6 . 3rd: Mezzanine card NIC IPv4 . 4rd: LOM Intel® I210 IPv6 . 5th: LOM Intel® I210 IPv4 . 6th: SATA HDD . 7th: SATA-CDROM 8th: Other removable device . If there is no bootable device found, BIOS should keep loop searching for bootable device. BIOS should support UEFI and legacy boot mode options and default is UEFI. UEFI and legacy boot mode have independent boot loop. Boot mode and boot order can be displayed and changed from BMC with OEM command.
6.3.4
Board ID
The motherboard should provide 4 strapping pins to be used as BOARD_ID[3:0], so BIOS can do correct board initialization based on different board configurations. Board ID is also routed to BMC to be accessed by BMC firmware.
6.3.5
FAB Revision ID
The motherboard should provide 4 strapping pins to be used as FAB_REVISION_ID [3:0], so BIOS can differentiate correct board FAB versions. FAB revision ID is also routed to BMC to be accessed by BMC firmware.
6.3.6
Remote BIOS Update Requirement
Vendors should provide tool(s) to implement remote BIOS update function. Vendor must validate update tools on each BIOS release during development and production. Tool(s) provided should support four update scenarios: Scenario 1: Sample/Audit BIOS settings o Return current BIOS settings, or . Save/Export BIOS settings in a human-readable form that can be . restored/imported (i.e. Scenario 2). Output must include detailed value- meaning description for each setting. Setting must include pre-production setup menus/options too. Scenario 2: Update BIOS with pre-configured set of BIOS settings Update/Change multiple BIOS settings. Setting must include pre-production . setup menus/options. Tool(s) should provide detailed value-meaning description for each setting. Reboot .
o o Scenario 3: BIOS update with a new revision Load new BIOS on machine and Update, retaining current BIOS settings . Reboot . Scenario 4: use BMC to update BIOS in PCH flash(also described in section 8.14) Update BIOS flash from BMC. . . Update need to be done with command line script in Linux environment from a remote server. Web GUI interface is not accepted. Additionally, the update tools and scripts should have the following capabilities: Update from the Operating System through ssh; the current OS based is CentOS 6.4 o 64-bit with updated Kernel specified by customer Require no more than one reset cycle to the system to complete BIOS update or BIOS o setup option change Require no physical access to system o BIOS update or BIOS setup option change should not take more than 5 minutes to o complete BIOS update procedure can be scripted and propagated to multiple machines o
6.3.7
Event log requirement
BIOS should do event log through BMC SEL with Generator ID 0x0001 and the combination of BIOS and BMC should meet the SEL log requirements in section 8.10.
6.3.8
BIOS Error Code Requirement
BIOS fatal error codes listed in following table should be enabled for POST CODE output. Vendor should display major and minor code alternatively as the requirement in 10.8.1.

6.3.9
POST Code for Option ROM Entry and Exit
Special BIOS post codes are assigned to indicate the entry and exit of option ROM. Two Bytes sequence are assigned for the entry and the exit to avoid same sequence used on other BIOS Post code. For example, use AA-Co indicates entry, and use BB-C1 indicates exit. These two sequences should be avoided to be used in other post codes process.
6.3.10 PPIN BIOS Implementation
BIOS shall support PPIN and set default to [Unlock and Enable]. There are two ways for user to access PPIN: BIOS shall map PPIN of CPUo and CPU1 to SMBIOS OEM Strings (Type 11) String . 5 and String 6. User can view PPIN value from Linux’s “dmidecode” command. BIOS shall implement Send_PPIN and Get_PPIN OEM Command to . communicate to BMC, per BMC’s request. User can retrieve PPIN information from BMC through OEM command. BIOS shall perform 2x actions to synchronize PPIN value to BMC: Serves SMI# signal from BMC and use  Send_PPIN OEM command to . communicate PPIN to BMC. . Use Send_PPIN OEM command to communicate PPIN to BMC when BIOS POST COMPLETE.
7
PCH Intel® SPS Firmware Plan of Record
Intel Motherboard V3.1 uses Intel® C610 series chipset. Its Management Engine (Intel® ME) runs Intel® Server Platform Services 3.0 (Intel® SPS 3.0) firmware. Intel® SPS firmware is required for system operation. Intel® SPS 3.0 Firmware (FW) consists of two parts, one required and one optional. Intel® SPS Silicon Enabling FW is required to boot the platform and have a fully functional platform. Intel® SPS Node Manager FW is optional. It provides power, thermal and compute utilization statistics, P/T-state control, simple and multi-policy power limiting at platform, memory, processor and core level with assist of a BMC. Intel® SPS firmware is stored in PCH flash. Vendor should provide utility to update Intel® SPS firmware in CentOS 6.4 64-bit with updated Kernel specified by customer through ssh. Utility should support updating Intel® SPS firmware and BIOS together or separately, and also provide option to update only Intel® SPS Firmware’s operation region or entire Intel® SPS firmware region. Vendor should also implement BMC to update PCH flash where Intel® SPS firmware is located as described in section 8.14.
8
BMC
Intel Motherboard V3.1 uses an ASPEED® AST1250 BMC for various platform management services and interfaces with hardware, BIOS, Intel® SPS firmware. BMC should be a standalone system in parallel to host (dual processor x86). The health status of host system should not affect the normal operation and network connectivity of BMC. BMC cannot share memory with host system. BMC management connectivity should work independently from host, and has no NIC driver dependency for Out-of-Band communication if using a shared-NIC.
8.1
Management Network Interface
BMC should have both I2C port and RMII/NCSI port for Out-of-Band access. Three options of Out-of-Band access methods should be supported. Option 2 and Option 3 share same device footprint as co-layout. Option 1: Shared-NIC uses I2C or RMII/NCSI interfaces to pass management traffic on data network of Mezzanine 10GE NIC. Option 2: Shared-NIC uses RMII/NCSI interfaces to pass management traffic on data network of Intel® I210-AS. Intel® I210-AS has SGMII/KX interface to mid-plane. Option 3: Shared-NIC uses RMII/NCSI interfaces to pass management traffic on data network of Intel® 1210-AT. Intel® 1210-AT has 10/100/1000 MDI interface to RJ45. BMC firmware needs to be flexible about which interface and device to activate by hardware strapping or a preset priority policy. BMC firmware needs to make sure the unused interfaces and devices are disabled and do not interfere with the activated management interface and device. MAC address of Out-of-Band should use NIC’s data network MAC plus an offset defined by NIC vendors. BMC management network firmware and utility need to support all features defined in this specification in both IPv4 and IPv6 network environment.
8.2
Local Serial Console and SOL
BMC needs to support two paths to access the serial console: A local serial console on debug header (described in section 10.8) and a remote console, also known as Serial-Over-LAN (SOL) through the management network (described in section 8.1). It is preferred that both interfaces are functional at all stages of system operation. During system booting, POST codes will be sent to port 80 and decoded by the BMC to drive the LED display as described in section 8.5. POST codes should be displayed in SOL console during system POST. Before the system has the first screen, POST codes are dumped to and displayed in the SOL console in sequence. For example, display as “[00] [01] [02] [E0]…” etc. After the system has the first screen in the SOL console, the last POST code received on port 80 is displayed in the lower right corner of the SOL console screen. A serial console buffer feature is required. A buffer needs to save at least the last 5x screens of local and 5x screens of remote console output; 80 column x24 row for each screen. The OOB raw command is used to extract and display the buffer. The buffer has to be stored in volatile media, such as internal or external SDRAM of BMC. SOL buffer data is cleared within 5 seconds of the removal of standby power. The SOL buffer should NOT be stored in any non-volatile media for security and privacy. SOL buffer implementation shall allow the SOL buffer being dumped by script with OEM command to a file (for scaling of data collection).
8.3
Graphic and GUI
Because the graphic user interface (GUI) is not scalable, the motherobard does not require BMC to support graphic, KVM and GUI features. All the BMC features need to be available in command line model by In-band and OOB IPMI command, or SOL.
8.4
Remote Power Control and Power policy
The vendor should implement BMC firmware to support remote system power on/off/cycle and warm reboot through In-Band or Out-of-Band IPMI command. The vendor should implement BMC firmware to support power on policy to be last-state, always-on, and always-off. The default setting is last-state. The change of power policy should be supported by IPMI command and take effect without BMC a firmware cold reset or a system reboot. It should take less than 3 seconds from AC on, for BMC to process power button signal and power up system for POST. A long waiting period from AC on, for BMC firmware to get ready before allowing system POST start is NOT allowed.
8.5
Port 80 POST
The vendor should implement BMC to support port 80 POST code display to drive 8 bit HEX GPIO to debug header. The BMC post function need to be ready before system BIOS starts to send 1st POST code to port 80. POST code should also be sent to SOL as mentioned in section 8.2. BMC should have access to POST code and record up to 256x POST codes. The OOB OEM command can be used to retrieve last 256x POST code from BMC.
8.6
Power and System Identification LED
The motherboard combines Power LED and System Identification LED to a single blue LED at front side. Power LED on is defined by the readiness of major run time power rails (P12V, P5V, and P3V3) but NOT the readiness of all run time power rails; for example, CPU core power rail being ready is not required for Power LED on indication. Power LED blinking is used as system identification. The on time is different during power on and power off. There are 4 states of Power/system identification LED depending on system power state, and chassis identify status.

8.7
Platform Environment Control Interface (PECI)
BMC should access Platform Environment Control Interface (PECI) through PCH SMLinko by default. PECI connection implementation should follow Intel guidelines. BMC should be able to execute PECI raw command by using Intel® ME as a proxy. Vendor should implement board design to connect CPU PECI interface to PCH PECI to be accessed by Intel® ME firmware directly by default, and reserve direct connection from BMC to CPU PECI interface as an option.
8.8
Power and Thermal Monitoring and power limiting
The vendor should implement BMC firmware to support platform power monitoring. To enable power limiting for processor, memory, and platform, Intel® SPS-NM is required. This function should be available through In-Band and Out-of-Band. The vendor should implement BMC FW to support thermal monitoring, including processor, memory, chipset, VRs, PCIe card, Mezzanine cards, Inlet/outlet air temperature, and airflow sensor. To make sure of temperature report accuracy, TI TMP421 with external transistor is preferred to be used for detect Inlet and Outlet temperature. Caution could be taken for inlet air sensor implementation to avoid preheats of nearby components, and reduce heat conducted through PCB. Airflow sensor is not a physical sensor, Airflow is calculated based on system FAN PWM.
8.9
Sensors
This chapter describes Analog, Discrete, and Event Only Sensors. The list includes all the sensors required. It does not include all the detail requirements. Please refer to sensor table below for more detail requirements.
8.9.1
Analog sensors
BMC has access to all analog sensors on the motherboard directly or through PCH Management Engine.  All analog sensors need to be displayed in sensor data record (SDR) repository. The analog sensors required are list as below. The lower and upper critical threshold is listed for system event logging purpose. Please refer to section 8.10 for logging requirements.

8.9.2
Discrete sensors
The vendor should implement BMC firmware access and display discrete sensors in SDR. BMC should log abnormal sensor reading to SEL. The discrete sensors required and the SEL format is listed as below for error decoding purpose. Please refer to BMC Sensor Table for more detail implementation requirement.

8.9.3
Event only sensors
The event only sensors are not shown in SDR. These sensors trigger SEL if abnormal value is detected. The event only discrete sensors required and Event Data 1,2,3 format is as the table below for decoding purpose. Generator ID in the event log shows which piece of firmware generates the log: 0x602C=Intel® SPS ME Firmware . 0X0001=BIOS/UEFI system Firmware . 0X0020=BMC Firmware . Please refer to BMC Sensor Table for more detail implementation requirement.

8.10 SEL
Vendor should implement BMC to support System Event Log (SEL). 8.10.1   Error to be logged . . Analog Sensors SEL is logged when Analog Sensor triggers upper or lower threshold. Discrete Sensors Sensor with “A” and “D” note in offset column means the related event will trigger assertion and de-assertion SEL. . . .
Memory Error
Both correctable ECC and uncorrectable ECC errors should be logged into SEL. Each log entry should indicate location of DIMM by CPU socket#, Channel # and slot #. Memory error reporting need to be tested by both XDP injection and reworked ECC DIMM. .
QPI error
All errors, which have status, register should be logged into Event Log. Fatal or non-fatal classification follows Chipset vendor’s recommendation. .
PCle error
All errors, which have status register, should be logged into Event Log, including root complex, endpoint device and any switch upstream/downstream ports if available. Link disable on errors should also be logged. Fatal, non-fatal, or correctable classification follows Chipset vendor’s recommendation. POST error . All POST errors, which are detected by BIOS during POST, should be logged into Event Log. Event Only Sensor The BIOS and BMC generated sensors defined in Event Only Sensor table is capable to trigger SEL. Notes below elaborates for some of the detail requirements for logging. Machine Check Error MCE shall be logged on running time when MCE generates SMI#. BIOS shall check MCE banks after a warm reboot and log error before clearing MCE .
Power error
Two kinds of power errors should be logged: SYS_PWROK FAIL: SYS_PWROK has a falling edge when SLP_S3_N is high; normal o AC/DC cycle/or HSC cycle shall not trigger this event PCH_PWROK FAIL = PCH_PWROK has a falling edge when SLP_S3_N is high; o normal AC/DC cycle/or HSC cycle shall not trigger this event
.
MEMHOT# , PROCHOT#  and VRHOT#
Memory hot error and Processor hot error should be logged. Error log should identify error source as internally coming from processor or memory, externally coming from voltage regulator, over-current triggered throttling, or under-voltage triggered throttling.
.
FAN Failure
FAN failure error should be logged if FAN speed reading is out of expected ranges between lower and upper critical threshold. Error log should also identify which FAN fails. .
PMBus status error
PMBus status sensors checks PMBus controller health status and log error if abnormal value is detected. PMBus controller can be a DC Hot Swap Controller (HSC) or PMBus AC to DC power supply unit. .
Intel® SPS FW related Error logging
For all above error logging and reporting, you may select to enable or disable each one of them based on needs.
8.10.2   Error Threshold Setting
. Enable error threshold setting for both correctable and uncorrectable errors, once programmed threshold is reached, an event should be trigged and logged. Memory Correctable ECC. Suggest setting threshold value to be [1000] in MP stage and [4] for evaluation, development, pilot run stage, with option of [1, 4, 10, 1000]. When threshold is reached, BIOS should log the event including DIMM location information and output DIMM location code through a debug card. ECC error event log threshold, defines the max number of correctable DIMM ECC is logged in the same boot. Default value is 10, with option Disable, 10, 50, and 100. QPI Error. Follow chipset vendor’s suggestion. PCIE Error. Follow chipset vendor’s suggestion. . . .
8.10.3   Critical SEL Filter
OEM commands are required to set and display two different level of SEL filtering. Default is to log all error during EVT/DVT/PVT with option to log only critical SEL that needs service or indicates power cycle state change, and SEL clear and overflow.
8.11  FSC in BMC
The vendor should enable Fan Speed Control (FSC) on BMC. The BMC samples thermal related analog sensors in real time. The FSC algorithm processes these inputs and drives two PWM outputs to optimized speed.

8.11.1   Data gathering for FSC
The sampling rate of the data should be x 1 sample/s.
8.11.2   FSC in BMC
Vendor shall follow and implement FSC and FSC update interface per Facebook Server Fan Speed Control Interface 7. The BMC should support both In-Band and Out-of-Band FSC configuration updates. Updates should take effect immediately without requiring a reboot.
8.11.3   Fan Connection
The motherboard has 2x FAN headers on motherboard. The motherboard and mid-plane interface also has optional FAN tachometer and PWM connections.
8.12  OEM commands
Vendor shall implement OEM features with OEM command listed in the table below
7 http://files.opencompute.org/oc/public.php?service=files&t=d48482b8b87a596dd93ac5b80e9fa3a2&download

8.13  BMC Firmware Update
Vendors should provide tool(s) to implement remote BMC firmware update, which will not require any physical access to the system. Remote update means either through Out- of-Band by management network or through In-Band by logging into local OS (CentOS) with data network. Tool(s) shall support CentOS 6.4 64-bit with updated Kernel specified by customer. A remote BMC firmware update may take a maximum of 5 minutes to complete. I2C sideband has bottle neck to achieve this requirement and NC-SI interface is needed. BMC firmware update process and BMC reset process require no reboot or power down of host system and should have no impact to normal operation of host system. BMC need to be fully functional with updated firmware after the update and reset without further configuration. In-band BMC firmware update can go through KCS or USB. USB is the preferred interface due to higher update speed. Default update should recovery BMC to factory default; option need to be provided to preserve SEL, configuration. MAC address is based on NIC MAC, so it would not be cleared with BMC firmware update.
8.14  BMC Update Dual PCH flash
8.14.1   Hardware Scheme
Vendor should implement BMC to be able to access host system PCH flash and recovery a host system PCH flash from corruption in remote. The PCH flash stores BIOS and Intel® SPS Firmware code; both BIOS and Intel® SPS Firmware region should be updated. A dual PCH flash chips hardware scheme is designed for this purpose. The hardware scheme contains two multiplexers controlled by BMC GPO. GPO_A controls PCH or BMC has access to flash. GPO_B controls primary or backup flash to be accessed by CS# signal. Primary flash is the default flash PCH is using to access BIOS and Intel® SPS Firmware code. Backup flash is stored with a known good image and usually is not modified. Backup flash can still be modified if needed by manual command line operations. . . . . The rules below should be followed to avoid unexpected system behavior: Default status of GPO_A and GPO_B shall be [0,0] during system AC on or during BMC booting and reset to make sure PCH has access to Primary Flash by default. BMC shall check system power status and not to change GPO_A and GPO_B status when system is in So. BMC shall set Intel® ME to recovery mode, before changing GPO_A and GPO_B status from [0,0] to another other value. BMC shall set Intel® ME to normal mode, after changing GPO_A and GPO_B status from other value to [0,0].

8.14.2 Software Scheme
The following software features must be supported: Auto BIOS recovery from backup flash to primary flash: 1) Whenﾕst boot up after BIOS update fails and the BMC watchdog timer (WDT) times out, the BMC determines that the BIOS needs to recover. It takes over control of flash access and duplicates backup flash content to primary flash. After that, BMC will return flash access to PCH and power cycle the system. This recovery attempt is only executed once after a failed BIOS update. BMC shall generate SEL for each recovery attempt. 2)    Manual BIOS recovery from backup flash to primary flash : User uses the IPMI OEM command to trigger BMC taking over control of flash access and duplicate backup flash content to primary flash. 3)    Manual BIOS save from primary flash to backup flash: User uses the IPMI OEM command to trigger BMC taking over control of flash access and duplicate primary flash to backup flash. 4)   Manual Mux control: User uses the IPMI OEM command to trigger the BMC to change the state of two multiplexers. 5)    Manual BIOS recovery from network to flash User uses the IPMI OEM command to trigger a PCH flash binary to be transferred to BMC and written to primary or secondary flash, depending on the state of the multiplexer. 6)    Enable/Disable Auto BIOS recovery User to use IPMI OEM command to Enable and Disable feature of Auto BIOS recovery; default is Disable for EVT/DVT/PVT and Enable for MP. 7)    Get dual BIOS feature status: User uses the IPMI OEM command to get the current state of two multiplexers, and auto recovery Enable/Disable status. 8)    Get dual BIOS version: User uses the IPMI OEM command to access and return BIOS version of both primary and secondary flash. Vendor shall provide an update utility that supports CentOS 6.4 64-bit with updated Kernel specified by customer.
8.15  BMC Update and Access CPLD
Vendor should implement BMC to access motherboard CPLD through JTAG and perform CPLD code upload from remote control server to BMC, update code from BMC to CPLD through CPLD JTAG interface, and verify CPLD code. All the steps above shall be done from the OOB command line mode. Vendor shall provide an update utility that supports CentOS 6.4 64-bit with updated Kernel specified by customer. BMC shall implement OEM command to read CPLD FW checksum, device ID, and FW version.
8.16  BMC Time Sync
. During BMC initialization: If it is 1st time AC on (G3 exit), BMC should get its initial RTC time from PCH RTC. Otherwise, BMC should keep its own RTC. . . . During BMC running time: If system is in So, and BIOS POST Complete, BMC should sync with PCH RTC every hour If system is in S5, BMC should retrieve the NTP UTC following OEM command setting of NTP periodical sync interval (default =2x per hour) . Log  system event (Sensor oxE9) if a mismatch of > ±5 seconds is found . Log NTP status time sync failed event (Sensor oxED) when BMC cannot obtain NTP time after 3x retries. Retry interval is set to 30 seconds by default.
8.17  PCIe and Mezzanine card Thermal monitoring
BMC should implement thermal monitoring feature for PCIe card on riser, and Mezzanine card. BMC reads the temperature of key components of PCIe and Mezzanine cards through its SMBus ports in the format as TMP421 temperature sensor. BMC uses the temperature reading in FSC and sensor reporting. Refer to “Add-on-Card Thermal Interface Spec for Facebook Servers” for detail requirement and implementation.
8.18  BMC PPIN Implementation
BMC shall send SMI# to PCH after BMC reset; BIOS shall serve the request by using Set_PPIN (OEM command) to write PPIN of CPUo and CPU1 to BMC.  BIOS also Set_PPIN when BIOS POST COMPLETE BMC shall return the PPIN to user with Get_PPIN (OEM command).
8.19  BMC Average Power Reporting
BMC shall record READ_EIN_EXT every 0.1 seconds in a ring buffer. BMC shall support OEM command Get_PIN to calculate and report the average power between current and a duration defined in the Get_PIN command. The ring buffer size shall support any duration from 0.1 second to 60 seconds, in 0.1 second increments. The return shall have a resolution of 0.1W.
8.20  BMC Access and Update VR Firmware
Vendor should implement BMC to access motherboard CPU VCCIN and memory VDDQ VR controllers’ firmware. Vendor shall implement a script or utility through the OOB master write-read command to perform a VR firmware code update and verify it from the OOB command line mode. The script or utility shall support CentOS 6.4 64-bit with updated Kernel specified by customer. During the VR firmware update, BMC and/or the update script shall stop the sensor polling to related VR. The VR firmware upgrade can be performed in S5. The capability to have a VR firmware upgrade in So is optional, and a DC cycle is allowed after a VR firmware upgrade in So to activate the new VR firmware. Vendor shall implement OEM command to read VR FW version.
8.21  BMC MSR Dump from OOB
Vendor shall implement BMC to dump MSR from both CPUs through Intel® ME and PECI. Vendor shall provide utility that supports CentOS 6.4 64-bit with updated Kernel specified by customer. This is a debug feature to allow the user to access critical debug information from faulty SUT on a server rack, without removing the system from a failure status and risking the loss of critical debug information. BMC firmware shall apply MSR dump automatically when there is IERR or MCERR. The BMC shall store the MSR dump in BMC flash. During the dump, the BMC shall reject chassis power related commands to avoid interrupting the dump.
8.22  BMC Network Status
. . . Vendor shall implement BMC to log a disconnection of both IPv4 and IPv6 oob network when the logging criteria is met. The logging criteria for sensor #0xB1, Event Data 1=00h is: BMC has been successfully assigned IP through DHCP with either IPv4 or IPv6 And, BMC cannot get response by pinging Default Gateway for IPv4 for consecutive 10x attempts; every attempt is 10 seconds apart And, BMC cannot get response by pinging Default Gateway for IPv6 for consecutive 10x attempts; every attempt is 10 seconds apart BMC shall not log the sensor #0xB1, Event Data 1=00h when either of IPv4 or IPv6 gateway has at least one response from gateway, in 10x consecutive pings. Only one Network Status log shall be logged by BMC until the next ping gets response on either IPv4 or IPv6 interface. BMC shall log sensor #oxB1, Event Data 1=01h for once when next ping gets response on either IPv4 or IPv6 interface.
Thermal Design Requirements
9
To meet thermal reliability requirement, the thermal and cooling solution should dissipate heat from the components when system operating at its maximum thermal power. The thermal solution should be found by setting a high power target for initial design in order to avoid redesign of cooling solution; however, the final thermal solution of the system should be most optimized and energy efficient under data center environmental conditions with the lowest capital and operating costs.  Thermal solution should not allow any overheating issue for any components in system. CPU or memory should not throttle due to any thermal issue under following environment. Inlet temperature lower than or equal to 35°C, and o inch H 2 O datacenter pressure . with all FANs in each thermal zone running properly . Inlet temperature lower than or equal to 35°C, and 0.01 inch H 2 O datacenter pressure with one FAN (or one rotor) in each thermal zone failed
9.1
Data Center Environmental Conditions
The thermal design for Intel Motherboard V3.1 needs to satisfy the data center operational conditions as described below.
9.1.1
Location of Data Center/Altitude
Data centers may be located 1000 meters above sea level or higher. Any variation of air properties or environmental difference due to the high altitude needs to be considered when creating the thermal design.
9.1.2
Cold-Aisle temperature
Data centers will generally maintain cold aisle temperatures between 18°C and 30°C (65°F to 85°F).  The mean temperature in the cold aisle is 24°C with 3°C standard deviation. The cold aisle temperature in a data center may fluctuate minutely depending to the outside air temperature of data center. Every component in system must be cooled and maintained below its maximum spec temperature in any of cold aisle temperature in a data center.
9.1.3
Cold-Aisle Pressurization
Data centers will maintain the cold aisle pressure to be between o” H₂O and 0.05” H₂O. The thermal solution of the system should be considered the worst operational pressurization in a data center, which it is o" H2O, and 0.01 "H2O with a single fan (or rotor) failure.
9.1.4
R.H
Most data centers will maintain the relative humidity to be between 20% and 85%. In the thermal design, the environmental condition changes due to the high altitude may not be considered when the thermal design can meet the requirement with maximum relative humidity, 85%.
9.2
Server operational condition
9.2.1
System loading
The power consumptions of individual components on the system motherboard will vary by  application or by motherboard SKU. The total power consumption of system also may vary with use or with the number of existence of PCIe cards on the system.  Please see summary below. System loading: idle to 100% . . Number of PCIe full height or half height cards can be installed: o to 3 . Number of PCIe Mezz card can be installed: o to 1 . Number of 3.5” HDD: 0 to 1 Plan of record worst case configuration for thermal and power delivery design is 2 x 145W TDP CPU with 16 x 16G DIMM. A unified thermal solution that can cover up to 100% system loading is preferred. However, an ODM can propose non-unified thermal solution if there is alternative way to provide cost benefits. At least the air-duct design should be unified for all SKU.
9.2.2
DDR DIMM DRAM Operation
Thermal design should meet DIMM max operating temperature as 85°C with single refresh rate. Thermal test should be done based on a DIMM module ‘s AVL (Approved Vendor List). The vendor should implement BIOS and memory subsystem to have optimized refresh rate and utilize optional DIMM Auto-Self-Refresh (ASR) based on DIMM temperature.  The implementation should follow updated DDR4 memory controller and DIMM vendor’s specification.
9.2.3
Inlet temperature
The inlet air temperature will vary. The cooling system should cover inlet temperatures as 20C, 25C, 30C, and 35C. Cooling above 30C is beyond operating specification, but used during validation to demonstrate design margin.  CPU throttling is not allowed to activate over the validation range oC – 35C.
9.2.4
Pressurization
Except for the condition when one rotor in server fan fails, the thermal solution should not be found with considering extra airflow from a data center cooling system. If and only if one rotor in server fan fails, the negative or positive DC pressurization can be considered in the thermal solution in the hot aisle or in cold aisle respectively.
9.2.5
Fan Redundancy
The server fans at N+1 redundancy should be sufficient for cooling server components to temperatures below their maximum spec to prevent server shut down or to prevent either CPU or memory throttling.
9.2.6
System airflow or volumetric flow
The unit of airflow (or volumetric flow) used for this spec is CFM (cubic feet per minute). The maximum allowable airflow per watt in the system must be 0.16.  The desired airflow per watt is 0.1 or lower in the system at the mean temperature (plus or minus standard deviation). See section 9.1.2 for the temperature definitions.
9.2.7
Delta T
The delta T is the air temperature difference across the system or the temperature difference between outlet air temperature of system and inlet air temperature of system. The delta T must be greater than 11.7°C (21°F). The desired delta T is 20°C (36°F) when the inlet air temperature to the system is lower than 30°C.
9.2.8
Thermal margin
The thermal margin is the difference between the maximum theoretical safe temperature and the actual temperature.  The board design operates at an inlet temperature of 35°C (95°F) outside of the system with a minimum 2% thermal margin for every component on the card. Otherwise, the thermal margin for every component in the system is at least 7% for temperatures up to 30°C.
9.3
Thermal kit requirements
Thermal testing must be performed up to 35°C (95°F) inlet temperature to guarantee high temperature reliability.
9.3.1
Heat Sink
The heat sink design should choose to be most optimized design with lowest cost. The heat sink design should be reliable and the most energy efficient design that satisfies all the conditions described above. The number of heat pipes in the heat sink should not be more than three. The ODM can always propose for different heat sink type if there is alternative way to provide cost benefits. The heat sink should be without complex installation guidance, such as air-flow direction.
9.3.2
System Fan
The system fan must be highly power-efficient with dual bearing. The propagation of vibration cause by fan rotation should be minimized and limited. The minimum frame size of fan is 60x60mm and the maximum frame size is 80x80mm. ODM can propose larger frame size of fan than 80x80mm if and only if there is alternative way to provide cost benefits. The maximum thickness of fan should be less than 38mm. Each rotor in the fan should have maximum 5 wires. Except for the condition when one fan (or one rotor) fails, the fan power consumption in system should not be exceeding 5% of total system power excluding the fan power. System fan should not have back rush current in all condition. System fan should have an inrush current of less than 1A on 12V per fan. When there is a step change on fan PWM signal from low PWM to high PWM, there should be less than 10% of overshoot or no overshoot for fan input current. System should stay within its power envelope (300W for Open Rack V1 configure) in all condition of fan operation.
9.3.3
Air-Duct
The air duct needs to be part of the motherboard tray cover, and must be most energy efficient design. The air-duct design should be simple and easily serviceable. The air-duct design should be unified for all SKUs. Using highly green material or reusable material for the air duct is preferred.
9.3.4
Thermal sensor
The maximum allowable tolerance of thermal sensors in the motherboard is ±3°C.
10 I/O System
This section describes the motherboard I/O requirements. 10.1  PCIe x24 Slot/Riser Card The motherboard has one PCIe x24 slot, which holds x24 PCIe Gen 3 signals from CPUo. Slot location must follow mechanical requirement that will be delivered in DXF format. . . The vendor should also provide two types of PCIe riser cards. 2 slots riser:  Top slot (slot3) is x16 PCIe connector with x16 signal and support full-height full-length PCIe card 8. Bottom slot (slot2) is x8 PCIe connector with x8 PCIe signal and open end and support full-height half-length PCIe card. 3 slots riser: Top slot (Slot4) is x8 PCIe connector with x8 signal and support full-height full-length PCIe card 9. Middle (slot3) and bottom (Slot2) slots are x8 PCIe connector with x8 PCIe signal and open end and support full-height half-length PCIe card. PCIe Riser card should fit to 2-OpenU height tray and replacing PCIe riser doesn’t need removal of any other components. Please follow Table 10-1 to use JTAG and reserved pins on PCIe x24 slot on MB to support two PCIe slots on riser card. Please follow Table 10-2 use reserved pins on PCIe x8 open end slot o (low) on riser card, and follow Table 10-3 to use reserved pins on PCIe x16 slot 1 (high) on riser card.
8 Full length card support needs a different heat sink on CPUO 9 Full length card support needs a different heat sink on CPUO
1 1 RSVD Riser card should implement SMBus Mux to avoid address confliction of PCIe cards. Follow Figure 10-1 and Figure 10-2 for implementation and slave address assignment in 8 bit. Figure 10-2 SMBus Mux on 3 slots riser . For 3 slots riser card, there are two additional requirements for power delivery: Generate P3V3 for riser slot from P12V on riser. P3V3 from motherboard shall not be connected to the riser card P3V3 regulator output. Add 2x2 right angle power connector (Molex/46991-1004 or equivalent) for additional P12V power from motherboard 8x pin ATX power connector to Riser card. More detail of motherboard 8x pin ATX power connector can be found in section 15.6. .

10.2  DIMM Sockets
The motherboard requires 15u” gold contact for DDR4 DIMM socket. Socket uses Yellow housing and white/nature latch for far end DIMM in a DDR channel, Black housing and white/nature latch for near end DIMM in a DDR channel. Vendor shall announce if the color selection will increase the cost of the DIMM Socket.
10.3  PCIe Mezzanine Card
Intel Motherboard V3.1 is compatible with Mezzanine Card for Intel v2.0 Motherboard20 and OCP Mezzanine Card 2.0 design specifications 11. The motherboard has one PCIe x8 Mezzanine Card Connector that holds x8 PCIe Gen 3 signal from CPUo. Mezzanine Card Connector on motherboard should use (FCI 61082- 121402LF) or equivalent connector, with mating (FCI 61083-124402LF) or equivalent connector on Mezzanine Card. Motherboard mezzanine connector is named as Slot1 in system.
10 http://files.opencompute.org/oc/public.php?service=files&t=2047e49112f6109c0f7e595cc93af8ae&download 11 http://files.opencompute.org/oc/public.php?service=files&t=b9b9b1892b8584c52aeaf53bf8706ce0&download

10.4 Network
10.4.1 Data network
The motherboard uses SFP+ 10G Mezzanine card as its primary data network interface at I/O side. There is option of single port or dual port. There is an x8 PCIe port routed to mid- plane connector for future data network option. One Intel® I210 1G NIC is placed on board to provide optional 10/100/1000 data connection.
10.4.2   Management Network
. . .
10.4.3   IPv4/IPv6 Support
The system need to have the capability to be deployed in both IPv4 and IPv6 network environment. All data network and management network should have this capability. The motherboard has 3x options of management network interface for BMC’s connection. Management network shares data network's physical interface. Management connection should be independent from data traffic, and OS/driver condition. The information below echoes the requirement in section 8.1, while emphasis on physical connection. SFP+ shared-NIC from Mezzanine 10G NIC or PCIe NIC, driven by BMC through RMII/NC-SI or I2C. I2C being default SGMII/KX shared-NIC connected to mid-plane interface from Intel® I210-AS, driven by BMC through RMII/NCSI 10/100/1000 MDI shared-NIC connected to RJ45 from Intel® I210-AT(co-layout with Intel® I210-AS), driven by BMC through RMII/NCSI This includes, but not limit to: DHCP and static IP setting, PXE booting capability, NIC and BMC firmware support, OS driver, and utility in both IPv4 and IPv6.
10.5 USB
The motherboard requires one external USB ports located in front of motherboard and one internal Type A USB connector. BIOS should support following devices on USB ports available on motherboard: USB Keyboard and mouse . . USB flash drive (bootable) . USB hard drive (bootable) . USB optical drive (bootable) The placement of internal Type A USB connector should consider to provide P5V to 1x SATA M.2 adaptor with 2280 size SATA M.2 card support.
10.6 SATA
The motherboard has Intel® C610 PCH on board. Intel® C610 has a SATA controller support 6x SATA3 ports, and an sSATA controller support 4x SATA3 ports. SATA Port 0~3 are connected to one vertical miniSAS connector.  sSATA Port 0~3 are connected to one miniSAS connector. SATA Port 4 is connected to 1x mSATA connector on motherboard by default. SATA Port 4 is also connected to M.2 connector with dual layout as backup. SATA Port 5 is connected to 1x vertical SATA connector on motherboard. Both mSATA connector and vertical SATA connector need to be placed near IO side of the motherboard for easy access. HDDs attached to all SATA connectors need to follow spin- up delay requirement described in section 15.6.3. The placement of vertical SATA connector should consider to provide SATA signal to 1x SATA M.2 adaptor with 2280 size SATA M.2 card support.
10.7 M.2
The motherboard has 1x optional onboard M.2 connector with Key ID=M and H4.2 Type. M.2 connector is populated only when mSATA connector is not populated as a co-layout. M.2 connector has option connection of PCIe x4 from PCH or SATA Port 4 from PCH. On board M.2 connector supports 2242, 2260 card form factor. Sideband signals such as WAKE# and SATA_ACTIVITY should be connected when it applies. For M.2 for factor 2280, the design should support 1x SATA to M.2 adaptor card. SATA signal is from vertical SATA connector on motherboard and power is from internal USB Type A connector.
10.8  Debug Header
The motherboard has one debug header placed in front of the motherboard. Debug card can be plugged into this header directly or through a cable. This debug header should support hot plug. Through this debug header, debug card should provide one UART serial port connector, two 7-segment LED displays, one reset button and one UART channel selection button. UART should provide console redirection function. Two 7-segment LED displays show BIOS POST code and DIMM error information. One reset button will trigger system reset when pressed. UART channel selection button sends negative pulse to motherboard to select and rotate UART console in a loop of host console-›BMC debug console->mid-plane console. Default after debug card plugged in or system power up is host console. A jumper is used to determine the power supply to debug header pin 14 is P5V or P5V_AUX. P5V is default and P5V_AUX is optional and used only for connecting BMC debug console and mid-plane console at S5. The jumper is for engineering use only, and located close to front of motherboard. The connector for the debug header is a 14 pin, shrouded, vertical, 2mm pitch connector. Figure 10-3 is an illustration of the headers. Debug card should have a key to match with the notch to avoid pin shift when plugging in.

10.8.1 Post Codes
The post codes are brought to debug header in HEX format via two HEX codes. The HEX codes can be driven by either the legacy parallel port (port 80) on SIO, or 8 GPIO pins. A debug card with two seven segment displays, two HEX-to-seven segment converters, logic level to RS-232 shifter and a RS-232 connector shall interface the debug header. During POST, BIOS should also output POST code to BMC SOL. So when SOL session is available during POST, remote console should show POST code as mentioned in section 8.2. During the boot sequence the BIOS shall initialize and test each DIMM module. If a module fails initialization or does not pass the BIOS test the following post codes should flash on the debug card to indicate which DIMM has failed. The first hex character indicates which CPU interfaces the DIMM module; the second hex character indicates the number of the DIMM module. POST code will also display error major code and minor code from Intel memory reference code. The display sequence will be “oo”, DIMM location, Major code and Minor code with 1 second delay for every code displayed. The BIOS shall repeat the display sequence indefinitely to allow time for a technician to service the system. DIMM location code table is as Table 10-11. DIMM number count starts from the furthest DIMM from CPU.

10.8.2 Serial Console
The output stage of the systems serial console shall be contained on the debug card. The TX and RX signals from the system UART shall be brought to the debug header at the chips logic levels (+3.3V). The debug card will contain mini USB type connector with pin definition shown in Table 10-12. A separate convertor is needed to provide RS-232 transceiver and DB9 connector.
5 GND The Debug card will contain a vertical receptacle female 6x pin 0.1" pitch header (FCI/ 68685-306LF or equivalent) to provide connection to an optional Class 2 Bluetooth module (Roving Networks/RN42SM²² or equivalent).

10.8.3 UART channel selection
When debug card is plugged, debug card pin 12 shall be used to notice motherboard BMC GPIO for its presence. When UART Channel selection button on debug card is pressed, same pin is used to send pulses to motherboard to trigger UART connection change among host console (Default) -> BMC debug console->mid-plane console and loop. Falling edge triggers change. Debug card has 10K pull down for pin 12, with a white UART Channel selection button between Pin 12 and ground. Motherboard side should implement logic to detect debug card presence when 10K or stronger pull down presents on Pin 12. Motherboard side should implement logic to trigger UART connection change when UART Channel selection button is pressed. UART channel selection has a power on reset value of 00, which means host console. Motherboard should stop its original POST code display for 1 sec when a falling edge to ground is detected, and give a 1sec display of Channel number to debug port POST code as an indication of UART channel change. If system POST code is not changed within this 1 sec, motherboard outputs original display. If system POST code is changed within this 1 sec, the latest POST code should be displayed. Motherboard also has two LEDs for displaying of UART connection status as described in Table 10-14.
10.8.4 Other debug use design requirements on motherboard XDP header is required for BIOS debug and should be populated in EVT and DVT samples. The access to the XDP header should not be mechanically blocked by CPU heat sink or other components. SMBUS debug header should be inserted for SMBUS on motherboard based on SMBUS topology vendor designs. SMBUS debug headers for PCH host bus and CPU/DIMM VR PMBus are required. If any other testing/debugging header is needed based on Intel platform development requirement, it should be added and populated in EVT/DVT samples.
10.9  Switches and LEDs
The motherboard shall include Power switch, Reset switch, Power LED, HDD activity LED and Beep error LED.
10.9.1 Switches
Vertical tactile switches are placed behind debug header. The push button actuator has a minimum 2.5mm diameter and protrudes 9mm+/-1mm from top of actuator to PCB surface. System power button is red and on the left. Systems reset button is black and on the right. If the power switch is depressed for durations less than four seconds a Power Management Event indicating that the power switch has been triggered shall be issued. If the power switch is depressed for durations longer than four seconds the motherboard shall perform a hard power off. If the reset switch is depresses for any duration of time the motherboard shall perform a hard reset and begin executing BIOS initialization code. Power switch and Reset switch function should not be gated by BMC FW or have dependency to BMC firmware readiness. The functionality of each switch shall be indicated by a label on the motherboard’s silkscreen. The labels PWR and RST are acceptable.
10.9.2 LEDs
The table below indicates the color and function of each LED. The motherboard’s silkscreen shall indicate the functionality of each of these LEDs. Silk screen labels are included in Table 10-15. Looking from I/O towards LED, from right to left, the sequence is Blue, Green, Yellow, Green, Green.

10.10  FAN connector
The motherboard has 2x system FAN connectors. FAN connector signals should follow “4- Wire Pulse Width Modulation (PWM) Controlled Fans Specification" Revision 1.3 September 2005 published by Intel Corporation. Each FAN has 6 pins, which is compatible with standard 4-Wire FAN connector, and also can be used to support dual rotor FAN that shares PWM control signal but has separate TACH signals. FAN connector pin definition is as in Table 10-16. LOTES YBA-WAF-016 or equivalent shall be used as Fan connector. Its mating part is a LOTES GAP-ABA-WAF-038 or equivalent. The fan power should be connected at the downstream side of hot swap controller (HSC). The fan power needs to be turned off during S5.

10.11  TPM Connector and Module
A 15pin vertical receptacle connector is defined on MB for LPC TPM module. Connector pin definition on motherboard side is shown in Table 10-17. FCI/91931-31115LF Receptacle or equivalent should be used on motherboard. Connector is shown in Figure 10-6. TPM module is a 23mm (L) x 13mm (W) x 1.57mm (T) PCB with FCI/91911-31515LF header or equivalent in the center of the module. Pin 1 2 3 4 5 6 7 8

10.12 Sideband Connector
An 8pin connector is defined for side band signals. Tyco/2-1734598-8 or equivalent is

11  Rear Side Power, I/O and mid-plane
11.1  Overview of Footprint and Population Options
There are 4x footprints at rear side of the motherboard to provide power to motherboard, and I/O to a mid-plane. The population of the footprints is flexible to fit the need of different use cases. Population options and major differences are listed in Figure 11-1. The high speed mid- plane is not covered in this document. The ORv1 implementation and low cost mid-plane is described in chapter 12 and section 11.3. The ORv2 implementation is described in

11.2  Rear Side Connectors
The mid-plane interface of the motherboard is described in this chapter to support 2x use cases. The motherboard has different stuff option to support both cases.  One mid- plane design is also described in this chapter to enable the motherboard to be used in Open Rack V1 in chapter 4.2.
11.2.1 Footprints and Connectors/Pressfit Cable
There are 3x types of connector footprints. A design can install a combination of 3x types of connectors and 1x type of Pressfit cable to the footprints. The AVL is listed in Table 11-1. The placement of connector footprints is shown in Figure 11-2.
Figure 1
11.2.2 AirMax® power 2x2
In use case with less or equal to 49 A ³³ on 12VDC, 1x pair of AirMax® power 2x2 R/A low profile headers and receptacle are populated. 2x pairs of such connector supports up to 98A on 12VDC.
11.2.2.1 Sled side
Up to 2x AirMax® power 2x2 R/A low profile headers are used at sled side, and shown in Figure 11-3. Pin definition is as Table 11-2. Figure 11-3 Power connector at sled side

11.2.2.2   Mid-plane side
Up to 2x AirMax® power 2x2 R/A receptacles for co-planar application are used at mid- plane side, and shown in Figure 11-4. Receptacle has long and short pin to control mating sequence. Part number with S-S-L-S pattern is used to ensure at least one ground pin mates before any power pin mates. Refer to Table 11-3 for detail. Figure 11-4 Power connector at midplane side

11.2.3 AirMax® Guide
1x pair of AirMax® 7.2mm R/A guide is used on the motherboard and mid-plane ONLY in the use cases that blind mate is needed. 11.2.3.1 Sled side Sled side uses 1x AirMax® 7.2mm R/A guide blade as Figure 11-5. 11.2.3.2 Mid- s 1x AirMax® 7.2mm R/A guide socket as Figure 11-6
11.2.4 AirMax® 3x8 signal
1x AirMax® VS/VS2 3x8 connectors follows the definition in Table 11-4 in the perspective of sled side. Signal description is shown in Table 11-5. It supports up to x8 PCIe plus side band signals and management interface. This connector is optional to support a high speed mid-plane with PCIe interface. Table 11-4 Pin Definition for AirMax® 3x8 signal

11.2.5   Pressfit Cable
A pressfit cable is enabled for the use case of ORv2 and Cubby chassis. A side view is shown in Figure 11-8. One side of the Pressfit cable is a pressfit power connector. The pressfit power connector is installed on motherboard with pressfit process, and secured by a screw for added strength. Pressfit power connector shares the same footprint and pin define as FCI/10124648-001LF. The other side of the pressfit cable is a panel mount connector. It is installed on a sheet metal panel with tool-less install and removal, shown in Figure 13-5. Pressfit cable is part of motherboard PCBA as a FRU; Pressfit cable cannot be replaced in the field.
11.3 Mid-plane
The motherboard design can support two types of mid-planes: Low cost mid-plane and high speed mid-plane. Low cost mid-plane is mid-plane with only power delivery, without high-speed signal routing on mid-plane or any high-speed signal switch on mid-plane. High speed mid-plane is mid-plane with power delivery, plus high speed interconnect on mid-plane. One or both of AirMax® 3x8 and AirMax® 3x6 need to be populated for this use case. Mid-plane can have one of these optional active components: high speed signal switch, high speed signal repeater/buffer. Mid-plane can have no active components as well, if mid-plane only provide high speed interconnect with PCB trace. Figure 11-9 intel Motherboard V3.1 to be used with Open Rack V1. For the use case shown in Figure 11-9. The motherboard is placed in a sheet metal tray similar to Winterfell tray 14 to form a sled for ORv1. Each sled includes a low cost mid- plane to bypass power from bus bar clip to motherboard. This mid-plane is completely passive and hot swap controller is on the motherboard. The interface between motherboard and mid-plane does not involve blind mating or hot swap. Hotswap interface is between bus bar clip and bus bar. Mid-plane provides mechanical and electric interface to DC power cable assembly described in section 12.3. Each of the two slugs of DC power cable assembly is fixed to mid-plane through 2x screws. There is a notch feature on lug of DC power cable. Mid- plane should design a key feature to mate with this notch to provide foolproof design. 2x 80mm FANs are directly attached to 2x fan connectors on motherboard. The design of mid-plane should allow the replace of mid-plane without removing motherboard from tray.
12  ORv1 Implementation
12.1  Shelf for ORv1
Figure 12-1 shows an ORv1 shelf for Intel Motherboard V3.1-ORv1 sled. Vendor should refer to 3D for more detail. ORv1 shelf provides mechanical interface between ORv1 rack and the sled. There is no electrical components in the shelf for ORv1.
12.2  Intel Motherboard V3.1-ORv1 Sled
A sheet metal tray serves as mechanical interface between motherboard and shelf. It also provides mechanical retention for the components inside the tray, such as bus bar cable assembly, fan, mid-plane, riser card, PCIe cards, Hard Drive, Mezzanine card. The combination of tray, motherboard and the other components assembled in the tray is an Intel Motherboard V3.1-ORv1 Sled.
12.3  Intel Motherboard V3.1-ORv1 Sled Power Delivery
There are 3x Bus bars in every ORv1 power zone. There are up to 3x sleds on the same shelf. Each sled is connected to bus bar directly with DC cable assembly and low-cost DC cable assembly includes bus bar clip assembly, DC cables, slug with screw hole and notch, and common mode choke.  Vendor is Methode, P/N is 5313-07450-00107, as shown in Figure 12-4. DC cable has two wires, red insulator for Power and black insulator for Ground to supply 12.5V DC power from bus bar clip to mid-plane.  DC cable is AWG6. The DC cable assembly, include including bus bar clip, DC cable and slug is rated for 50A in worst operation condition. The sled has a low cost midplane. One power connector is installed by default and support up to 49A of continous current in 65C local ambient. Midplane is passive connection without capacitive loading. Hotswap controller on motherboard provides soft start of P12V on the motherboard and avoid inrush to bus bar.
13  ORv2 Implementation
13.1 Cubby for ORv2
Figure 13-1 shows Cubby enclosure for Intel Motherboard V3.1-ORv2 sled. Vendor should refer to 3D for more detail. Cubby serves as the mechanical and power delivery interface between ORv2 and Intel Motherboard V3.1-ORv2 sled.
13.2  Intel Motherboard V3.1-ORv2 sled
A sheet metal tray serves as mechanical interface between the motherboard and Cubby. It provides mechánical retention for the components inside the tray, such as pressfit cable, fan, riser card, PCIe cards, Hard Drive, Mezzanine card. The combination of tray, motherboard with pressfit cable and the other components assembled in the tray is an Intel Motherboard V3.1-ORv2 sled. Vendor should refer to 3D for more detail.
13.3  Intel Motherboard V3.1-ORv2 Power Delivery
There is one bus bar in each power zone of ORv2. There are up to 3x sleds in each cubby enclosure. delivery to each of the 3x sleds. Medusa cable delivers up to 40A to each node continuously in 65C local ambient considering pre-heating from sleds. One side of Medusa cable has a bus bar clip assembly to interface with bus bar. The other side of Medusa cable has 3 spited panel mount connectors with built in mechanical floating feature. Figure 13-4: Medusa Cable drawing top view Each motherboard for ORv2 sled has a pressfit cable installed on motherboard. Detail of the pressfit cable is described in section 11.2.5. One side of the pressfit cable is a pressfit connector that installed on the motherboard; the other side of the pressfit cable is a panel mount connector (shown in Figure 13-5) that interfaces with one of the panel mount connectors on medusa cable. Panel mount connector can be removed from the sled without using tools.
14 Mechanical
Intel Motherboard V3.1-ORv1 sled and Intel Motherboard V3.1-ORv2 sled should work with Open Rack V1 and ORv2 mechanically and the implementation guide provided in Chapter 12 and Chapter 13.
14.1  PCIe and HDD bracket
There is a metal bracket near I/O side of the tray to provide mechanical support for two full-height PCIe cards and a 3.5" hard drive or three full-height PCIe cards.
14.2  Fixed Locations
Refer to mechanical DXF file for fixed locations of mounting hole, PCIe x16 slot and power connector.
14.3  PCB Thickness
To ensure proper alignment of the motherboard and mid-plane interface within its mechanical enclosure, the motherboard should follow PCB stack up in Table 5-3 to have 85mil (2.16mm) PCB thickness. And mid-plane PCB thickness should also be 85mil (2.16mm). Mezzanine card and riser card PCB thickness should be 62mil (=1.6mm).
14.4  Heat Sinks and ILM
The motherboard shall support heat sinks that are mounted according to the Intel’s E5- 2600 v3 thermal mechanical specification and design guide. The vendor shall comply with all keep out zones defined by Intel in the above referenced specification. Standard LGA2011-3 heat sink and Narrow ILM solution is preferred. Only when vendor's CPU/VR placement doesn’t allow standard Intel heat sink and Narrow ILM to be used, the vendor may choose non-standard heat sink, however this may require approval from the entity purchasing the motherboard.
14.5 Silkscreen
The silkscreen shall be white in color and include labels for the components listed below. Additional items required on the silkscreen are listed in section 16.3. CPUo / CPU1 . . DIMM slot numbering, as described in Figure 10-4 . LEDs as defined in 10.9.2 . Switches as PWR and RST
14.6  DIMM Connector Color
Colored DIMM connectors shall be used to indicate the first DIMM of each memory channel. This first DIMM on each channel is defined as the DIMM placed physically furthest from its associated CPU. This DIMM connector shall be populated first when the memory is only partially populated. The First DIMM connector shall be a different color than the remaining DIMM connectors on the same memory channel.
14.7  PCB Color
Different PCB colors shall be used to help identify the motherboards revision. Table below indicates the PCB color to be used for each development revision.

15  Motherboard Power system
15.1  Input Voltage
15.1.1 Input voltage Level
The nominal input voltage delivered by the power supply is 12.5 VDC nominal at light loading with a range of 11V to 13V. The motherboard shall accept and operate normally with input voltage tolerance range between 10.8V and 13.2V when all under voltage related throttling features are disabled in section 15.2. Motherboard’s under-voltage protection level should be less than 10.1V.
15.1.2 Capacitive Load
Previous server generations required a maximum capacitive load of 4,000 uF. This requirement does not apply to Intel Motherboard V3.1 design. The motherboard design requires greater than 10,000 uF capacitive loading on P12V_AUX for supplying surge current from CPU VR, and reduce the slew rate of P12V_AUX decaying for NVDIMM feature at surprising power fail. Hot Swap Controller design should limit the inrush current to node during soft-start to less or equal to 10A.
15.1.3   P12V as AUX rail
There is only one 12V rail delivered to the motherboard as auxiliary power rail. Caution need to be taken to provide proper isolation to PCIe device, HDD, FAN, and all other devices in system, to meet voltage and timing requirement during running time and power on/off. The isolation circuit should have a soft start to avoid inrush current to P12V Aux rail, and prevent SOA damage of isolation MOSFET.
15.2  Hot Swap Controller (HSC) Circuit
In order to have a better control of 12.5V DC power input to each motherboard, one HSC (ADI/ADM1278) is used on the motherboard. HSC circuit provides the following functions: Inrush current control when motherboard is inserted and powered up. . Current limiting protection for over current and short circuit. Over current trip point . should be able to set to 41.4A and 48.8A with jumper setting; default is 48.8A. HSC UV protection shall be set to 10V~10.1V . . SOA protection during MOSFET turning on and off. HSC fault protection is set to latch off (default) with retry as stuff option. . PMBUS interface to enable PCH Intel® ME and BMC following actions . Report server input power and log event if it triggers upper critical threshold. o Report input voltage (up to 1 decimal point) and log event if it triggers either o lower or upper critical threshold. Log status event based on hot swap controller’s status register. o Use HSC or external circuit to provide fast (<20us) over current sense alert to trigger . system throttling and CPU fast PROCHOT#; feature need to be controlled by BMC GPIO directly. BIOS has a setting to control Enable/Disable/ [no change]. No change is the default. This means follow the BMC initial setting. BMC sets it to disable as the default. Before BMC is ready, the hardware POR state is enable. Use HSC or external circuit to provide fast (<20us) under-voltage alert to trigger system . throttling and CPU fast PROCHOT#. This feature is enabled by default with resistor option to disable. . Use HSC or external circuit to provide fast (<20us) under-voltage alert to trigger system FAN throttling. This feature is disabled by default with resistor option to enable. Use HSC or external circuit to provide HSC timer alert to trigger system throttling before . HSC OCP happens. Please refer to Table 15-1 for setting requirements of System, CPU, and memory sub- . system throttling.
. . The voltage drop on HSC current sense resistor should be less or equal to 25mV at full loading. Hot swap controllers should have SMBUS address set to 0x11 (7bit format). The power reporting of hot swap controller needs to be better than 2% from 50W to full loading in room temperature.
15.3  CPU VR
15.3.1 CPU Maximum Power
The motherboard shall be designed to handle a processor with a maximum TDP of 145W for Intel® Xeon Processor E5-2600 v3 Product Family. Support for processors of higher TDP is not required. As a result the vendor shall optimize the CPU VR accordingly.
15.3.2 CPU VR Optimizations
CPU VR optimizations shall be implemented to remove cost and increase the efficiency of the power conversion system. Vendors shall only use the minimum number of total phases to support the maximum CPU power defined in 15.3.1. CPU VR should have auto phase dropping feature, and run at optimized phase count among 1, 2, 3,..., and maximum phase count. CPU VR should support all Power States to allow the VRM to operate at its peak efficiency at light loading CPU VR should be compliant to latest VR12.5 specification and validation method and pass test with margin.
15.3.3 CPU VRM Efficiency
. . . . . . . . For CPU efficiency measurement: VID is set to 1.7V Vin is set to 12.5V Efficiency is measured from input inductor to socket Driver and controller loss should be included Output voltage is gathered from Vsense of socket No additional air flow shall be supplied to the VR area other than the air flow caused by VRTT tool FAN Test is done in room temperature(20°C~25°C) Voltage measurement shall be done by tool and method with 0.05% accuracy or better 15 Based on ADM1278 Timer threshold=1V for over-current protection
Current measurement shall be done by tool and method with 0.25% accuracy or better . Efficiency curve shall be higher than the envelope defined below Table 15-2 Vendors are encouraged to exceed the above efficiency requirement and may propose higher efficiency VRMs that may come at additional cost. Power efficiency measured from 12.5V input to CPU socket should also be analyzed and improved.
15.3.4 CPU core VR configuration
Vendor should use CPU core VR solution with all configurations stored in NVRAM without any external resistor strapping. Vendor should provide utility under CentOS to perform VR configuration change. Configuration change should take effect without AC cycling node. The guaranteed rewrite count of NVRAM should be greater or equal to 15.
15.4  DIMM VR
15.4.1   DIMM Maximum Power
The motherboard has a DIMM configuration of 2 CPU sockets, 4 channels per socket, and 2 Slot per channel. Vendor should follow memory controller vendor’s guideline to design and validate DIMM power rail to support maximum power needed for this configuration, and support 1.2V DDR4 DIMM.
15.4.2 DIMM VR Optimizations
DIMM VR should support auto phase dropping for high efficiency across loading. DIMM VR should be compliant to latest VR12.5 specification and memory controller vendor's updated validation guideline, and pass test with margin.
15.4.3 DIMM VR Efficiency
For DIMM VR efficiency measurement VID is set to 1.20V . Vin is set to 12.5V . Efficiency is measured from input inductor to PCB near DIMM sockets . Driver and controller loss should be included . Output voltage is gathered from PCB at middle of the 2 nd and 3 rd of 4x DIMM slots . No additional air flow shall be supplied to the VR area . . Test is done in room temperature(20°C~25°C) Voltage measurement shall be done by tool and method with 0.05% accuracy or better . Current measurement shall be done by tool and method with 0.25% accuracy or better . Efficiency curve shall be higher than the envelope defined below . Table 15-3

15.4.4 DIMM VR configuration
DIMM VR has same configuration requirement as CPU VR, listed in 15.3.4.
15.5  VRM design guideline
For VRM, vendor should list current budget for each power rail based on worst case loading case in all possible operation conditions. General requirements for VR component selection and VR design should meet 150% of this budget, and OCP should set to 200% of this budget. Vendors should do design check, inform purchasers about the actual OCP setting chosen for VRM and explain the reason if it cannot meet this general requirement above. For VRM which requires firmware or power code or configuration file, vendors should maintain version control to track all the releases and changes between each version, and provide a method to retrieve version through application software during system run time. This software method should run under CentOS 6.4 64-bit with updated Kernel specified by customer. All switching VRs should reserve testing hook for bode plot measurement.
15.6  Hard Drive Power
The motherboard shall supply power to all possible 9 hard drives connected. Drives require 12VDC and 5VDC power sources. For 1 individual SATA ports, power will be delivered through a traditional 4-pin floppy disk power connector, Tyco 171825-4 or equivalent. The mating connector is a Tyco 171822-4. The pin assignment shall follow industry standard convention described in Table 15-4. Table 15-4 4-pin floppy disk power connector
For SATA ports inside the miniSAS connector, power will be delivered through an 8-pin (2X4) ATX power connector shown in Figure 15-3. Connector is Foxconn/HM3504E-EP1 or equivalent. Pin definition is described in Table 15-5. It fans out to be 8 standard SATA power cables. Power design should allow this connector also be used to supply P12V on PCIe riser. Figure 15-3

15.6.1 HDD Power Requirements
The motherboard must provide enough power delivery on 12.5VDC and 5VDC to support up to 9 HDDs this platform supports. This means to support 1A continuous per HDD on 12.5VDC power rail, and 0.75A continuous per HDD on 5VDC power rail. In-rush current required to spin up the drive must also be considered in power delivery design.
15.6.2   Output Protection
Both 12V and 5V disk output power rails shall protect against shorts and overload conditions.
15.6.3 Spin-up Delay
When hard drive spins up after power on, it draws excessive current on both 12V and 5V. The peak current may reach 1.5A ~ 2A range in 12V. System may have up to 9 hard drives installed, so there is need to spin up hard drive in sequence. BIOS should implement 5 seconds delay between each hard drive spinning up. In order to do this, SATA hard drive's power cable should have pin 11 as NC (No Connection) to enable hard drive's spin-up delay function.
15.7  System VRM efficiency
Vendors shall supply high efficiency VRMs for all other voltage regulators over 20W not defined in this specification. All other voltage regulation modules shall be 91% efficiency over the 30% to 90% load range. Vendors are encouraged to deliver systems with higher efficiencies. If higher efficiencies are available at additional cost vendors shall present those options.
15.8  Power On
Motherboard should be set to restore last power state during AC on/off. This means that, when AC does on/off cycle, motherboard should power on automatically without requiring power button. Only when motherboard is powered off on purpose, then motherboard should be kept power off through AC on/off.
16  Environmental and Regulations
16.1 Environmental Requirements
The motherboard shall meet the following environmental requirements: Gaseous Contamination: Severity Level G1 per ANSI/ISA 71.04-1985 . . Ambient operating temperature range: -5C to +45C . . Operating and Storage relative humidity: 10% to 90% (non-condensing) Storage temperature range: -40C to +70C Transportation temperature range: -55C to +85C (short-term storage) . The full system shall meet the following environmental requirements: Gaseous Contamination: Severity Level G1 per ANSI/ISA 71.04-1985 . . . Ambient operating temperature range: -5C to +35C Operating and Storage relative humidity: 10% to 90% (non-condensing) Storage temperature range: -40C to +70C . . . Transportation temperature range: -55C to +85C (short-term storage) Operating altitude with no de-ratings: 1000m (3300 feet) System would be deployed into datacenter with following environment. Site 1 as Temperature: 65F to 85F . . Humidity: 30% to 85% . Altitude: 1000m (3300 feet) Site 2 as Temperature: 65F to 85F . . . Humidity: 30% to 85% Altitude: 300m (1000 feet)
16.2  Vibration & Shock
The motherboard shall meet shock and vibration requirements according to the following IEC specifications: IEC78-2-(*) & IEC721-3-(*) Standard & Levels, the testing requirements are listed in Table 16-1. The motherboard shall exhibit fully compliance to the specification without any electrical discontinuities during the operating vibration and shock tests. No physical damage or limitation of functional capabilities (as defined in this specification) shall occur to the motherboard during the non-operational vibration and shock tests.

16.3 Regulations
Vendor need to provide CB reports of the motherboard and tray in component level. These documents are needed to have rack level CE. The sled should be compliant with RoHS and WEEE.  The motherboard PCB should have UL 94V-0 certificate. The vendor should design an EMI panel kit and pass FCC Class A.
17  Labels and Markings
The motherboard shall include the following labels on the component side of the motherboard. The labels shall not be placed in a way, which may cause them to disrupt the functionality or the air flow path of the motherboard.

18 Prescribed Materials
18.1 Disallowed Components
. . .
18.2  Capacitors & Inductors
. . . . Tantalum capacitor using manganese dioxide cathode is forbidden. SMT Ceramic Capacitors with case size › 1206 are forbidden (size 1206 still allowed when installed far from PCB edge, and with a correct orientation that minimizes risks of cracks) . Ceramics material for SMT capacitors must be X7R or better material (COG or NPo type should be used in critical portions of the design), with the exception of X6S can be used in CPU Cage area. Only SMT inductors may be used. The use of through-hole inductors is disallowed.
18.3  Component De-rating
For inductors, capacitors and FETs, de-rating analysis should be based on at least 20% de- rating.
19 Reliability and Quality
19.1 Specification Compliance
Vendors must ensure that the motherboard meets these specifications as a stand-alone unit and while functioning in a complete server system. The vendor is ultimately responsible for assuring that the production motherboards conform to this Specification with no deviations. The Vendor shall exceed the quality standards demonstrated during the pilot build (PVT) while the motherboard is in mass production. Customer must be notified if any changes are made which may impact product quality.
19.2  Change Orders
Vendors must notify customer any time a change is made to the motherboard. A Specification Compliance Matrix will be submitted to customer for each revision of the motherboard including prototype samples.
19.3  Failure Analysis
The following components shall not be used in the design of the motherboard. Components disallowed by the European Union’s Restriction of Hazardous Substances Directive (RoHS) Trimmers and/or Potentiometers Dip Switches The following limitations shall be applied to the use of capacitors. Only Aluminum Organic Polymer Capacitors shall all be used they must be rated 105C, and shall be selected only from Japanese Manufacturers. All capacitors will have a predicted life of at least 50,000 hours at 45C inlet air temperature, under worst conditions. Vendors shall perform failure analysis on defective units, which are returned to the vendor. Feedback shall be provided to customer with a Corrective Action plan within two weeks from the date, which the units were received at Vendor's Facility.
19.4 Warranty
The Vendor shall warrant the motherboard against defects and workmanship for a period of two years from the date of initial deployment at customer’s facility. The warranty is fully transferable to any end user.
19.5  MTBF Requirements
The motherboard shall have a minimum calculated MTBF of 300K hours at 90% confidence level at 45C ambient temperature. The motherboard shall also demonstrate the MTBF requirement above by running at full load and 50% of time and performing AC cycling test 50% of time at 45C. Typical alternation period is 1 week for stress test and 1 week for AC cycling test. This MTBF demonstration shall finish prior to First Customer Shipment (Pilots samples, Mass Production units). The motherboard shall have a minimum Service Life of 5 years (24 Hours / day, Full Load, at 45C ambient temperature). Vendors shall provide a calculated MTBF number based on expected component life.
19.6  Quality Control
. . . Below is a list of manufacturing requirements to ensure ongoing product quality: Incoming product must have less than 0.1% rejections Cpk values will exceed 1.33 (Pilot Build & Production) Vendors will implement a quality control procedure during Production, by sampling motherboards at random from the production line and running full test to prove ongoing compliance to the requirements. This process shall be documented and submitted prior to Production. The relative reports shall be submitted on an ongoing basis. . Vendors will conduct an ongoing burn-In procedure for use in Production (Production will not start without an agreement on some sort of burn-in procedure). Vendors shall submit documentation detailing the burn in procedure.
19.7 Change Authorization and Revision Control
After the motherboard is released to mass production, no design changes, AVL changes, manufacturing process or materials changes are allowed without prior written authorization from customer. The AVL (Approved Vendor List) is defined by the list of components specified in the BOM (Bill of Materials). Any request for changes must be submitted to customer with proper documentation showing details of the changes, and reason for the changes. This includes changes affecting form, fit, function, safety, or serviceability of the product.  Major changes in the product (or in the manufacturing process) will require re-qualification and/or re- certification to the Product. A new set of First Article Samples may be required to complete the ECO process. Any modifications after approval shall phase-in during production without causing any delays or shift of the current production schedule. Vendors shall provide enough advance notice to customer to prevent any discontinuation of production. All changes beginning with the pilot run must go through a formal ECO process. The revision number (on the motherboard label) will increment accordingly. Revision Control: copies of all ECOs affecting the product will be provided to customer for approval.
19.8  PCB Tests
Server ODM should arrange Independent 3 rd party lab testing on SET2DIL, IST, and IPC- 6012C for each motherboard, riser card and mid-plane PCB from every PCB vendors. Mid- plane without high speed differential signal does not require SET2DIL test. Server ODM cannot use the PCB vendor for these tests. Server ODM should submit reports for review and approval before a PCB vendor can be used in mass production. The testing lots should be manufactured at the same facility of a PCB vendor with same process that planned to be used by mass production. SET2DIL requires 5x different PCB fabrication lots from the PCB vendor.  Environmental shipping, packaging, and handling of this board is vital to test success; overnight shipping direct from PCB vendor to SET2DIL independent lab is recommended. IST is done once. It is required to be tested on a board manufactured at the same time as a board that completely passes SET2DIL. (Run SET2DIL, if it passes then ask the IST lab to run IST on the board they receive.) IST test profile is 3x cycles to 250°C and up to 1000x cycle to 150°C. Passing criteria is 150x cycles average, and 100x cycles minimum for 35x coupons. IPC-6012C is done when 2x of the 5x SET2DIL tests passing from a PCB vendor. (Passing at the independent test lab) ODM should work with PCB house to implement IST and SET2DIL coupon to break off panel without increasing unit cost of PCB.
19.9  Secondary Component
Secondary component planning should start from EVT and reach 80% of total number of BOM items in PCBA BOM in EVT. The rest of secondary component should be included in DVT. It is recommended that PCB is planned with 3 vendors at EVT. EVT and DVT build plan should cover all possible combinations of key components of DC to DC VR including output inductor, MOSFETs and driver. ODM should provide 2nd source plan and specification compare before each build stage.
20 Deliverables
20.1  OS Support
Motherboard shall support CentOS 6.4 64-bit with updated Kernel specified by customer, and pass Red Hat certification tests.
20.2 Accessories
All motherboard related accessories, including heat sink, back-plate and CPU socket protectors, should be provided and installed at the vendor’s factory. All accessory boards including debug card, PCIe riser card, should be provided by the vendor.
20.3 Documentation
The vendor shall supply the following documentation to customer: Projection Action Tracker . . Bug Tracker . Testing Status Tracker . Design documents Schematics for EVT, DVT and PVT(Cadence and PDF) o Board Layout EVT, DVT and PVT (Cadence and Gerber RS-274) o Board Design Support Documents: o System Block Diagram . Power distribution Diagram . . Power and Reset Sequence Diagram . High Speed Signal Integrity Simulation, especially for DDR4 memory . Power Integrity Simulation, for important power rails such as CPU and DDR4 memory . SMBUS and JTAG Topology . GPIO Table for BMC and PCH . Hardware Monitor Topology . Clock Topology . Error Management Block Diagram BIOS Version plan, Version Tracker, and specification BMC Version plan, Version Tracker, and specification . . . BMC Sensor Table . . . . . Mechanical 2D Drawings (DXF and PDF) Mechanical 3D model (IGS or STEP, and EASM) BOM with MFG name, MFG P/N, Quantity, Reference Designators, Cost BOM in customer’s defined format, whose definition is provided in separate file. Validation documents o    Server Hardware Validation Items: Test Plan and Report FAI test plan and Report o VR test Plan and Report o Signal Integrity Test Plan and Report o Functional Test Report o MTBF Test Plan and Report, including calculation o System AVL(CPU, DIMM, PCIe cards, Mezzanine Cards, SSD) Qualification Test Plan o and Report Reliability Test Plan and Report o De-rating Report (worst conditions) o 2nd source component Plan and Test Report o Thermal Test Plan and Report (with indication of critical de-ratings, if any) o Mechanical Test Plan and Report o
20.4  Mass Production First Article Samples
Prior to final project release and mass production, the Vendor will submit the following samples and documentation: . . . . . . .
21 Shipping
The motherboard shall be shipped using a custom packaging containing multiple motherboards in each package. The quality of the packing assembly will be such that the motherboard will not get damaged during transportation. The units shall arrive in optimum condition and will be suitable for immediate use. Shock Test for the shipping box shall be conducted by the Vendor and submitted to customer for audit and approval.
22 Appendix
22.1  Appendix: Commonly Used Acronyms
This section provides definitions of acronyms used in the system specifications. GPIO – general purpose input output ANSI – American National Standards Institute l²C – inter-integrated circuit BIOS – basic input/output system IPMI – intelligent plattform BMC – baseboard management management interface controller KCS – keyboard controller style CFM – cubic feet per minute LAN – local area network (measure of volume flow rate) LPC – low pin count CMOS – complementary metal– oxide-semic on ductor LUN – logical unit number DCMI – Data Center Manageability MAC – media access control Interface MTBF – mean time between failures DDR4 – double data rate type 4 MUX – multiplexer DHCP – dynamic host configuration NIC – network interface card protocol 00B – out of band DIMM – dual inline memory module ORv1 – Open Rack Version One DPC - DIMMs per memory channel ORv2 – Open Rack Version Two DRAM – dynamic random access memory OU – Open Compute Rack Unit (48mm) ECC – error-correcting code PCB – printed circuit board EEPROM - electrically erasable programmable read-only memory PCle – peripheral component interconnect express EMI – electromagnetic interference PCH – platform control hub FRU – field replaceable unit
All the pertinent documentation described in section 20.3 and any other documents and reports, necessary for customer to release the product to mass Production. Pilot samples which are built in the allocated Facility for mass production. A full Specification Compliance Matrix A full Test/Validation Report Production line final Test ‘PASS’ tickets Samples which have passed the production burn-in process Samples shipped using the approved for production-shipping box described in section 21. POST – power-on self-test PSU – power supply unit PWM – pulse-width modulation PXE – preboot execution environment QPI – Intel® QuickPath Interconnect QSFP – Quad small form-factor pluggable RU – rack unit (1.75") SAS – serial-attached small computer system interface (SCSI) SATA – serial AT attachment SCK – serial clock SDA – serial data signal SDR – sensor data record SFP - small form-factor pluggable SMBUS – systems management bus SMBIOS – systems management BIOS SOL – serial over LAN SPI – serial peripheral interface SSD – solid-state drive SSH – Secure Shell TDP – thermal design power TOR – top of rack TPM – trusted platform module U – Rack unit UART – universal asynchronous receiver/transmitter UEFI – unified extensible firmware interface UL – Underwriters Laboratories
22.2  Mechanical drawings
. Following mechanical drawings are provided 3D CAD for Intel Motherboard V3.1-ORv1 sled 3D CAD for Intel Motherboard V3.1-ORv12 sled (preliminary, subject to design change) .
22.3  SMBIOS FRU mapping table
22.4  Add-on-Card Thermal Interface Spec for Intel Motherboard V3.1