Computer Servers Draft 2 Comment Response Summary

Ref. #

Organization

Topic

1

Comment Summary

Resilient server definition

Comment Summary

Two stakeholders commented on the product family definitions. The firststakeholder appreciates EPA's approach to certifying a server family byidentifying the range of configurations which meet the ENERGY STAR active
efficiency and idle power thresholds and that the range of certified configurations
EPA thanks stakeholders for the support of the product family structure, and has added
can be a subset of the total configurations which can be created for a given
additional hardware specific guidance (both socket and memory focused) to better ensure
server product family.
Product Family definition
that minimum and maximum performance configurations across different product families
The second stakeholder supports defining the capabilities of product family using are constructed in a similar way.
three configurations, but is not clear on how EPA expects a product family to beassessed. They recommend that a sub-section be added to the specification toclarify how companies should handle certification of a subset of a product family
and certification of a single configuration.

2

3

Comment Summary

4

Comment Summary

5

Comment Summary

6

Comment Summary

APA Definition

Stakeholder Comment Summary

EPA Response

One stakeholder commented on the resilient server definition. The stakeholder
commented that most, but not all, resilient servers will no longer use memory
EPA is proposing a revised resilient server definition in Draft 3 based on significant industry
buffer chips. They suggest including a requirement that resilient servers are able feedback, and welcomes feedback on the changes.
to provide a maximum memory capacity of 1.5 terabytes.

Two stakeholders commented on the APA definition. The first stakeholder statedthat it is important to define expansion and integrated APAs. The stakeholders
also suggested changes to the APA definition to ensure the distinction between
After extensive discussion with stakeholders, EPA is maintaining the previously proposed
these two APA types is accurate.
definitions of integrated and expansion APAs proposed in Draft 2. In addition, due toincreasing complexity in diverging advanced APA technology approaches and having no way
One of the two stakeholders also recommends the APA definition be expanded
to test the performance of APAs in the test method, EPA is proposing that computer servers
with two sub-definitions to accommodate APA implementations beyond GPUs.
which can support expansion APAs shall be tested and certified without them. Computer
The second stakeholder supports this differentiation as well, and the exclusion ofservers with integrated APAs continue to remain out of scope in Draft 3.
integrated APAs from the V3 server requirements. This stakeholder encourages
EPA to work with industry to determine the best approach for managing
integrated APAs in V4.

Two stakeholders support the consolidating product categories from five to three. After reviewing the latest data set, the proposed active state efficiency requirements in Draft
Computer Server Product
The first stakeholder also encourages EPA to revise categories to better align3 are comprised of 11 categories. This granularity is required to adequately differentiate
Categories
with performance.
computer servers with varying performance levels and architectures.
I/O Definition

One stakeholder asks EPA to clarify whether an internal RAID/SAS controller can The existing definition of I/O device states that RAID/SAS controllers are an example of I/O
be considered an I/O device.
devices for purposes of the specification.

HPC Definition

Two stakeholders commented on the High Performance Computing definition.
The first stakeholder proposed several minor revisions to the definition. The
second stakeholder stated that the first stakeholder's revisions were importantbecause of HPC's rapidly advancing sophistication, allowing HPC tasks to be
performed with a smaller number of heterogeneous nodes.

EPA has adopted the proposed changes to the HPC definition in Draft 3 and welcomesstakeholder feedback on the changes.

7

Comment Summary

Data Collection

One stakeholder commented on their ongoing data collection efforts. They stated
Between Draft 2 and Draft 3, EPA did allow industry to gather additional data on newer
that they have been able to get a reasonable sample of two socket rack products,
computer servers released in 2017, and used those data points in the level setting analysis
but request an extension to April 2018 to collect SERT active efficiency and idle
for Draft 3. Given the changes in the market and the types of products that could bepower data from additional products. They also stated that it may be appropriate
expected to be certified over the lifetime of the specification, EPA has utilized data received
to remove server products from the 2013 and 2014 launch years to better match
between 2014 and November 2017.
the group of servers likely to be on the market in 2018/2019.

8

Comment Summary

New Processors

Two stakeholders commented that a next generation of products will be released
soon by both AMD and Intel. The second stakeholder will gather data for theseSee comment #7 above.
processors so that EPA can assess the value of modifying the proposed active
efficiency limits or delaying the release of Draft 3

Computer Servers Draft 2 Comment Response Summary

Four stakeholders commented on EPA's proposal to set both active state and idle
state efficiency criteria. Three stakeholders strongly supported EPA's proposal.

910Comment Summary

Comment Summary

The fourth stakeholder conducted an analysis that indicates that when comparing After considerable discussion with stakeholders, and evidence provided by Green Grid andaccepted by stakeholders, which shows that idle state power is adequately captured within
deployed power, the combined metric with performance adjusted idle and pure
Active and idle efficiency
active efficiency limits perform significantly better than the EPA Draft 2 combined the active state testing within SERT, EPA is proposing to move forward in Draft 3 with active
requirements
metric using idle without the system performance adder. In addition, the average state efficiency requirements only. Due to this change in approach, the active state
requirements proposed in Draft 3 are notably more aggressive than those found in Draft 2.
idle power of the configurations which passed the metric is 12% higher of theservers that pass the Draft 2 metric. This suggests that, even if the passed
servers operate in idle all of the time, the servers that passed the active metricwould have the lowest total power use because fewer servers would be deployed.

Active efficiency
requirement

Two stakeholders commented on the active efficiency requirements. The firststakeholder stated that a separate active efficiency limit should be established for As noted in Comment #4 above, EPA has expanded the number of categories in the active
one and two socket servers, because the proposed limit will be more severe on state efficiency requirements to 11, differentiating by socket count as well as product type.
one socket servers. This stakeholder also stated that EPA should use SERT
V2.0.0 as the server active efficiency metric. The second stakeholder encouraged EPA has adopted SERT V2.0.0 and used the data set converted to SERT V2.0.0 values forlevel setting purposes in Draft 3.
EPA to carefully assess the revised V2.0.0 active efficiency scores and ensurethat the thresholds are set appropriately for resilient servers.

11

Comment Summary

Adders for APAs

One stakeholder proposed an idle adder for APAs, storage devices and I/OIdle allowances for non-APA hardware have been removed since idle state efficiency
devices. They stated that they would provide additional data and input for these requirements are no longer found in Draft 3.
adders on Oct 16, 2017. This stakeholder supports that the allowance of 30 watts
for each APA device, but wants to make the distinction that the idle limit should be The separate APA idle allowance of 30 watts proposed in Draft 2 has been removed due tocomplexity explained in comment #3 above.
set per device.

12

Comment Summary

APA Idle Limit

One stakeholder commented on the APA idle limit. This stakeholder does notbelieve the proposed 30 watt idle limit for GPU expansion cards is tooconservative for server GPUs.

See comment #3 above.

13

Comment Summary

This stakeholder proposes exempting high-end expansion APAs from scope. The
stakeholder recommends EPA sets exclusion for any APA with a local memory
High-end expansion APAs bandwidth of greater than 700 GB/sec. This same stakeholder previously stated See comment #3 above.
that they were undertaking efforts to find idle power data on expansion APAs
because they believed the 30 watt maximum value was set too low.

14

Comment Summary

One stakeholder stated that it is appropriate to revise both active and idle
Efficiency requirement for
efficiency requirements for volume servers because these newer products havevolume servers
higher idle power, performance, and active efficiency values.

15

Comment Summary

Internal Power Supply
Efficiency Test

EPA has included newer data from computer servers released in 2017 to inform the active
state efficiency levels proposed in Draft 3.

One stakeholder commented that the reference to Generalized Internal Power
EPA has updated the second reference of this protocol to rev. 6.7 to match the first
Supply Efficiency Test Protocol refers to two different revisions and suggests thatreference in Draft 3.
they should be aligned.

16

Comment Summary

Memory capacity
requirements

Two stakeholders commented on memory capacity requirements. The firststakeholder recommends that EPA set a minimum set of requirements for the
As noted above, EPA has added both memory capacity and socket population guidance inmemory capacity of the low-end and high-end configurations, and provides
suggested requirements for the configurations. The second stakeholder supports the product family definition.
the open memory capacity requirements for the 3 test configurations that define
the product family.

17

Comment Summary

Power Management
Reporting

One stakeholder commented that the Power and Performance Sheet on the
ENERGY STAR website needs to be updated.

EPA will ensure that all ENERGY STAR computer server websites are up to date.

Computer Servers Draft 2 Comment Response Summary

Three stakeholders commented on the power supply requirement. Two
stakeholders recommended that EPA increase low-load energy efficiency
requirements, due to the fact that servers often share load on redundant power
supplies, and may spend most of their operating lives loaded to 10-20% rated
capacity.

18

Comment Summary

Power Supply
Requirement

The second stakeholder stated that the average server power supply spends
most of its time in 3-20% load range, and recommends revising the power supply
Industry provided evidence during the March 12 stakeholder meeting showing that typicalefficiency requirements to better represent real-world working conditions. They
computer server power supply loads do not drop below 10%. Draft 2 already contained astate that a 4% improvement in PSU efficiency a low levels results in a 4%
requirement for the 10% load for single-output power supplies, and EPA does not have datasystem-level efficiency gain. This stakeholder also requests that EPA sets a 10%
to support setting a 10% load requirement for multi-output power supplies. Therefore, EPA isload requirement for PSUs, and that this requirement is 86%. Finally, they
proposing to continue harmonizing with the 80Plus Platinum levels in Draft 3.
recommend aligning the power factor requirement of .9 at 50% to the 80-PLUS
Platinum requirement of .95
One stakeholder agreed that EPA should consider giving greater weight to low
load points. They stated that the efficiency requirements for PSUs at the ten andtwenty percent load points above the current levels of 83% and 90% is worth
consideration. They also would like EPA to consider a requirement at ten percent
for mutli-output and supply units.

Stakeholder requested clarification if the 80Plus standard report woudl be
sufficient for a worldwide valid server certificate.

19

Comment Summary

Power Supply
Requirement Question

In addition, the stakeholder requested clarification as to the voltage the power
supply should be tested at for a global valid server certificate adn if there aredefined efficiency requirements indenpendent from the test voltage.

For ENERGY STAR certification, internal power supplies are only required to meet the 115V
power supply efficiency requirements. In order to list markets for products outside the U.S.,
the products are required to be tested and meet the efficiency requirements at all relevant
voltages.

The stakeholder also noted that the efficinecy requirements are easier to maintainat 230V than for 115V.
An 80Plus test report is sufficient for EPA ENERGY STAR certification.

20

Comment Summary

Power Supply
Requirement Question

One stakeholder asked if the platinum certified power supply 80Plus standard test
The 10% load requirement for single-output power supplies does not harmonize with thereport is sufficient for a worldwide valid server certificate. They also asked the80Plus Platinum requirement, because there is no 10% load requirement for 80Plus
voltage required to test for a global valid certificate, and if the defined efficiency
Platinum. Rather it is an increase in the 10% load required in Version 2.0, with a raise inrequirements are independent from the test voltage. The stakeholder also statedmagnitude similar to the increases at the 20%, 50%, and 100% load points from Version 2.0
that the 83% efficiency at 10% load required for a single-output PSU does not
to Version 3.0.
match the 80Plus platinum requirement, and asks for the origin of thisrequirement. They also state that power factor requirements in table 2 do not
Similarly, the ENERGY STAR power factor requirements are more granular that what ismatch the platinum requirement.
provided at 80Plus, but the requirements do align where possible (0.95 @ 50% load forsingle-output).

Computer Servers Draft 2 Comment Response Summary

21

Comment Summary

SERT conversion

Two stakeholders commented on SERT conversion. One stakeholder
commented that the changes to the calculation of the flood and capacity memory
worklet performance values mean that SERT activity efficiency scores will change
based on the memory capacity of the given configuration. In order to set active
efficiency thresholds, it will be necessary to convert V1.1.1 scores to V2.0.0. This
stakeholder is working to complete this conversion. Once this conversion iscomplete, this stakeholder states that it will be necessary to reset the active
As noted above, EPA has adopted SERT V2.0.0 and worked with the adjusted data set for
efficiency and idle power thresholds to ensure a 25% passing rate. Thislevel setting purposes in Draft 3.
stakeholder does not feel that the modifications to the memory worklet score
calculations have affected the ability of the SERT test to assess servers forenergy efficiency
The other stakeholder stated that additional analysis is required to determine
whether EPA analysis of server data is capable of representing active serverenergy performance. This stakeholder states that so far they have seen an
insufficient basis for belief that this is so.

22

Comment Summary Sleep power management

EPA is open to the idea of incentivizing this behavior, but with the removal of idle state
Two stakeholders proposed a provision and incentive for very low-power, "sleep"
requirements is unsure of a tangible way to reward the behavior. EPA welcomes additional
state servers.
stakeholder feedback on a pathway forward in this area.

23

Comment Summary

EPA has clarified in the product family definition that product families certified and shipped as
One stakeholder commented on the requirement that UUTs have all processor
sockets populated during testing. They asked if this means that tests of a low-end ENERGY STAR must match the populated socket count used in testing, which according toperformance configuration with one CPU are not allowed by servers with more
the requirements in Section 6.1.2 means that in most cases all sockets must be populated
than one processor socket.
for ENERGY STAR purposes.

Test methods

24

Comment Summary

GeneralOne stakeholder stated that the data underlying Draft 2 of this specification has
not been revealed that demonstrates that servers certified to this specification
would provide cost-effective energy savings to customers without compromising
performance. The stakeholder requests that EPA shares the data, analysis, andassumptions it used to create this specification.

25

Comment Summary

GeneralOne stakeholder stated that there are two main opportunities for reduction inserver energy consumption in data centers: ensuring servers operate efficiently
and consolidating lightly-loaded servers into fewer servers.

26

Comment Summary Storage device categories

One stakeholder suggests that EPA create 6 categories of storage devices withdifferent idle allowances.

27

Comment Summary

Component Idle Adders

One stakeholder commented that there is a wide variation in idle power between
and within storage devices due to differences in form factor, device speed or
SSD, communications protocol used and capacity. The stakeholder suggests anidle allowance for storage components for this reason

Idle adders for higher
throughput I/O Ports

One stakeholder collected additional data on the idle power of I/O ports anddetermined the following:
The 8W idle limit for >10 Gbit active ports should be extended to less than 50
Gbit.
An additional category, greater than or equal to 50 Gbit and less than or equal to
100 bit should be added with a value of 13 W per active port.
Ports with over 100Gbit of throughput should be excluded from certification andverification testing because they are specialty ports.
Testing of in-scope servers can be done without installing over 100 Gbit Ethernet
cards.

28

Comment Summary

29

Comment Summary

30

Comment Summary

Idle allowance for network One stakeholder recommends creating four additional network port categories
portswith idle allowances
Memory Adder

One stakeholder thanks EPA for reducing the idle state efficiency adder limits for
additional memory above 4GB of Installed Memory to .125 W/GBEPA is releasing a public data set with the release of Draft 3 with all the information pertinent
to level settings and determine energy usage of proposed ENERGY STAR vs. non-ENERGYSTAR products.

EPA agrees with this statement and is planning to update its buying guidance and the
ENERGY STAR website to provide more up to date tips for end users. Any stakeholderinterested in working with the Agency should reach out to ENERGY STAR to express thisinterest.

Computer Servers Draft 2 Comment Response Summary

31

32

Comment Summary

Comment Summary

Idle Power Allowances

One stakeholder commented that on Table 5, line 492, the footnote numbering
does not match with the referring explanation starting on line 450.

I/O Adders

Two stakeholders commented on I/O adders. The first stakeholder requests toincrease the active port adder for 40Gbit+ network cards. The second stakeholder EPA thanks stakeholders for their comments on the idle allowances. As noted in comment
also proposes additional adders for higher capacity ports. Both stakeholders#9 above, EPA has removed idle state efficency requirements from Draft 3. EPA held
support EPA's proposal for ports up to 10 Gb/s.
lengthy discussions with stakeholders, including two meetings, to discuss if SERT Version

Memory adderdiscussions resulted in agreement across stakeholders that the active state efficiency metric
One stakeholder commented that EPA's proposed memory adder is too
adequately incorporates and drives idle state efficiency as well. As such, the idle
restrictive. This stakeholder requested that EPA share the data set used todetermine this memory allowance. This stakeholder commented that the memory requirements were removed from the specification, thereby precluding the need for idle
adder in W/GB should be set with consideration to memory technology and DIMM allowances. Although the active state metric was used to set ENERGY STAR levels
size. They also propose an adder for new memory technologies. The stakeholder identifying the top quartile of the market, EPA will continue to provide idle information on theis concerned that establishing strict idle adders will result in delays or prohibition product finder and certified product list for those stakeholders interested in that information.
of more efficient, high powered memory technologies.

2.0.0 sufficiently incorporates idle test results within the overall active state metric. These

33

34

Comment Summary

Comment Summary

Memory Adder

One stakeholder commented that the memory adder is far higher than required bymainstream technology. They state that half of the models on the QPL have morethan 64GB of memory, and a quarter have 300GB. They propose to use a
hyperbolic tangent equation to show that power draw does not scale linearly withcapacity, or, alternatively to implement a 2-step linear allowance.
This stakeholder conducted an analysis of the additional memory adder of .125
watts _ AdditionalMem(GB) beyond 4 GB of installed memory and found that thisapproach resulted in higher allowances than justified by the calculations. This ineffect provides a giveaway to high-end configurations

35

36

Comment Summary

Comment Summary

Idle requirement

System Performance
Adder

One stakeholder encourages EPA to set a low idle level with an adder for highperformance servers. This would allow better differentiation on the basis of idle
and active state efficiencies, which still allowing efficient high-performanceservers to meet the specification
One stakeholder commented on the need for a system performance adder. Thisis especially important due to the recent release of the new AMD and Intelprocessors. This stakeholder proposed new base idle and system performance
multiplier values to respond to EPA's concerns and better balance the number ofhigh performance systems removed due to the idle limits.
The stakeholder suggests that the adder be assessed based on CPU Peak
Performance and system performance multiplier.