Method to display real-time module OIR time

In one embodiment, a device in a modular networking rack determines an online insertion and removal (OIR) time for a particular device module of a plurality of device modules contained within the modular networking rack. The device determines an OIR time for the plurality of device modules based on the OIR time for the particular device module and provides an indication of the OIR time for the plurality of device modules for display to a user.

TECHNICAL FIELD

The present disclosure relates generally to computer networks, and, more particularly, to a method to display real-time module online insertion & removal (OIR) time.

BACKGROUND

In a modular configuration system, online insertion & removal (OIR) is a process of replacing a faulty device module without disrupting the performance of the entire system. During this process, the system continues to be functional, and a faulty device module (e.g., a line card, a power supply, a fan tray, etc.) is replaced with a new one. In most cases, while a faulty device module is being replaced, the system is exposed to the external environment through the opening created by the missing device module. This can cause the system airflow to be unbalanced, as this empty space would offer a least resistance path and airflow would bypass through this space causing other areas and/or components to not get enough airflow for cooling.

Among all the device modules to be replaced, a fan-tray OIR is typically the most critical. Notably, if there is no redundant fan tray in the system, the system would not have any airflow for its cooling, and there could be system reliability issues due to overheating. Therefore, fan-tray OIR should be completed as soon as possible to avoid system failure. The time limit for completing the OIR would depend on the overall system configuration and traffic patterns, system ambient conditions (temperature, altitude etc.), and thermal behavior of the system.

Currently, systems are being designed to provide sufficient OIR time based on an estimated duration for device module replacement. Such a time limit is often published in hardware installation guides. However, published times are valid only for a particular system configuration at a particular ambient temperature. Any change in system configuration, such as due to an upgrade, often invalidates the published time-limits. Furthermore, many installations have shifted from an ambient temperature of 23° C. in the 1990s to 30° C. ambient temperature today as a means to save on massive electrical bills.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

According to one or more embodiments of the disclosure, a device in a modular networking rack determines an online insertion and removal (OIR) time for a particular device module of a plurality of device modules contained within the modular networking rack. The device determines an OIR time for the plurality of device modules based on the OIR time for the particular device module and provides an indication of the OIR time for the plurality of device modules for display to a user.

DESCRIPTION

As noted above, online insertion and removal (OIR) is a process in which a faulty or out of date device module can be replaced without affecting system performance (e.g., within a networking rack/chassis). This process is similar to hot swapping in some ways. However, hot swapping is primarily a hardware function and generally requires no software commands, whereas OIR is typically a software feature. Notably, special software instructions are required to be executed during OIR, both before removing a device module and after installing a replacement device module. During this procedure, an identical device module should be used to replace the original one. In addition, when performing OIR on multiple device modules or components within a device (e.g., a router), the operation should typically be performed on one device module at a time.

Generally, fan tray OIR (e.g., the replacement of a faulty or out-of-date fan tray) is a special situation which should be treated carefully. For example, typically a system may be designed to operate with all fans working in a fan tray. In some cases, the system may still be capable of operating with a single fan fail, but the fan tray should be replaced with a new one at the earliest convenience. In addition, new high performance fans are being developed which provide better cooling performance, and it may be desirable to upgrade an existing fan tray with a new high performance fan tray by performing an OIR.

However, if a fan tray OIR is not completed within a prescribed time, the entire system may shut down in order to avoid overheating and damage to the device modules or components. Thus, special software (SW) monitoring is generally required during fan tray OIR to ensure that a catastrophic failure of components due to high temperatures is avoided. In other words, fan tray OIR needs to be completed within a prescribed time limit that may depend at least on the operating temperature. This would provide an estimate of the time to reach failure limit once the fan tray is removed from system.

An example plot of fan tray OIR time is illustrated by the time-temperature graph shown inFIG. 1. As shown, a system (e.g., a modular networking rack) including a fan tray may be operating at temperature110, which may be relatively constant over the period of time prior to device replacement. During a fan tray OIR, initiated at time115, the fan tray may be removed, thereby changing the conditions within the system. With no (or significantly reduced) air flow, the temperature of the system would be expected to rise, as illustrated by slope120. If the temperature reaches maximum permissible temperature130, system failure may occur. Thus, it is important to know the time-limit to failure140and to monitor the OIR process time, to be certain the time-limit is not exceeded.

Method to Display Real-Time Module OIR Time

The techniques described herein introduce several intelligent methods to determine device module OIR time in real-time and, further, to provide a visible indication (e.g., a display) of the determined OIR, thereby alerting a user (e.g., operator initiating the OIR) of the amount of time available for the device module OIR process. In this way, the user is able to avoid any network down time due to system shutdown (e.g., caused by system overheating or thermal runaway during a fan tray OIR). A variety of approaches are provided to determine device module OIR for a plurality of device modules contained within a modular networking rack and to present the OIR time information to a user. In some embodiments, an intelligent software process is used to monitor the real-time device temperature and/or system airflow for a fan tray OIR. Several mechanisms are also introduced herein to leverage the resulting information, to display the real-time OIR time-limit.

Specifically, according to one or more embodiments of the disclosure as described in detail below, a device in a modular networking rack determines an online insertion and removal (OIR) time for a particular device module of a plurality of device modules contained within the modular networking rack. The device determines an OIR time for the plurality of device modules based on the OIR time for the particular device module and provides an indication of the OIR time for the plurality of device modules for display to a user.

Illustratively, the techniques described herein may be performed by hardware, software, and/or firmware, such as in accordance with an OIR time process, which may include computer executable instructions executed by a processor, to perform functions relating to the techniques described herein.

Operationally,FIG. 2AandFIG. 2Billustrate the thermal behavior of two different device modules, “Device-A” and “Device-B” respectively, which may be considered to be the thermally worst device modules for a particular modular networking rack system (e.g., device modules that are the most sensitive to fan tray removal) during a fan tray OIR. Assuming that the system is working in steady state, Device-A and Device-B would each have their own operating temperatures,210A and210B, depending, for example, on the type of device and its physical location within the modular networking rack. These temperatures, as well as other relevant conditions (air flow rate, power levels, etc.) may be monitored by a controlling device in the system. Both device modules would also be expected to have different power dissipation properties as well as different values of maximum permissible temperature,230A and230B, and different heating characteristics in no-airflow conditions (as illustrated by slopes220A and220B). These may be known based on the device type or determined by a controlling device in the system.

As soon as a fan tray is removed from a modular networking rack containing Device-A and Device-B, these devices will start to heat up, as per their characteristic curves. In order to avoid the temperature reaching the maximum limit (e.g., maximum permissible device temperatures230A and230B), the system may be configured to power down at a temperature below these limits, if the fan-tray is not inserted back in the system in time. If the time taken by the system to initiate the shutdown until complete power down is known (e.g., time to shutdown245A and245B), a safe operating point for the devices can be identified (e.g. temperature235A and235B). This safe operating point determines the particular device OIR time-limits,250A and250B, thereby providing sufficient time to initiate system shutdown without damaging the devices.

Upon determination of each particular device module OIR time, an OIR time for the plurality of device modules in the modular networking rack may also be determined. In particular, the OIR time for the various device modules may be compared. For example, the worst available time (e.g., the lowest device OIR time) for any device may define the OIR time limit for the system. For example, as shown inFIG. 2AandFIG. 2B, Device-B has a shorter time limit than Device-A. Therefore, the system OIR time limit may be defined by Device-B. Alternatively, consideration may also be given to the importance or sensitivity of each device module to temperature increases. If a fan tray is inserted back at the “safe operating point” for Device-B, the temperature of the system would begin to ramp down, and the system would return to its normal operating condition.

Once the device (e.g., system controller) determines an OIR time for the plurality of device modules within the rack, based on the OIR time determined for one or more individual device modules of the plurality, there are several ways this real-time information can be provided to a user. In one embodiment, the OIR time process may provide an indication of the determined system OIR time to an electronic display. For example, such a display may be a system console, LCD display on the networking rack/chassis itself, or the like. Alternatively, command line interface (CLI) commands can be published in hardware install guides, to get real time data from device running the OIR time determination process.

In one embodiment, the device/controller may output an indication of the determined OIR time for the plurality of device modules to an LED via a flash pattern. The LED may be positioned anywhere in the system, including, for example, on an interior surface of the rack or on one of the devices within the rack (e.g., a system controller, a controller card, a fan tray, or the like). For example, as shown inFIG. 3, device module300in may include LED310capable of producing a flashing pattern or other visible indication of the available OIR time as an option. In order to interpret the LED flash information for the OIR time, label320may be included on device300, including, for example, flash color and/or duration and corresponding time to system shutdown), and this information may also be included in hardware install guides.

As shown inFIG. 3, the indicated flash pattern may only be configured to provide time-range information (e.g., between 2-3 minutes) rather than and an exact time-limit. Sometimes, for higher power density systems, it may be necessary to know the exact time available before system shutdown so that the device module replacement can be completed safely within the prescribed time.

Thus, in some embodiments, a time-code flash pattern may be used to provide a visual indication to a user of an exact time available for device module OIR (e.g., via one or more LEDs). For example, as shown inFIG. 4AandFIG. 4B, single LED410A and410B positioned on device module400A (which may be a fan tray, as shown) or400B, respectively, may be provided in order to display the determined system OIR time code information using a flashing pattern and/or color to represent a number. Alternately, as shown inFIG. 4C, multiple LEDs411C,412C, and413C, may flash sequentially or simultaneously, with each single LED representing a numerical digit of the system OIR time. Different LED color schemes (e.g., to represent the severity as the system approaches shutdown) can also be used.

A variety of different flash patterns (color, frequency, duration, etc.) may be used as an indication of the system OIR time. In particular, different LED colors may be used to provide numerical information of the overall OIR time. For example, if the determined OIR time for a plurality of device modules in a networking rack has 3 digits (minutes and seconds), each may be represented by a single LED (e.g., single LED410A shown inFIG. 4A) as follows:a green flash with a 1 second interval—first numerical digita solid green for 2 seconds—pausea green flash with a 1 second interval—second numerical digita solid green for 2 seconds—pausea green flash with a 1 second interval—third numerical digita solid green for 5 seconds—pause before starting next cycle

In addition, the flash pattern of a single LED may further include different LED colors along with varying flash intervals to display the OIR time to a user. For example, to display a 3 digit number for a system OIR time limit (in seconds), each digit may be represented by a single LED (e.g., single LED410B shown inFIG. 4B) as follows:a green flash with a 2 second interval—the first numerical digit,a yellow flash with a 2 second interval—the second numerical digit, anda red/blue flash with a 2 second interval—the third numerical digit.

Multiple LEDs may provide additional flexibility and clarity for displaying OR time code information. In particular, as shown inFIG. 4C, each LED (411C,412C, and413C) may be assigned to represent the position of numerical digit in the OIR time number value. For example, LED411C may flash a pattern to represent the first numerical digit, LED412C may flash to represent the second numerical digit, and LED413C may flash to represent the third numerical digit. As a specific example, a time of 458 seconds may be displayed as follows:first LED—4 flashes with 2 second intervals (total time=8 seconds)pause for 2 seconds (total time=10 seconds)second LED—5 flashes with 2 second intervals (total time=20 seconds)pause for 2 seconds (total time=22 seconds)third LED—8 flashes with 2 second intervals (total time=38 seconds)pause for 2 seconds (total time=40 seconds)

Thus, the entire time code may be displayed to a user within 40 seconds, and this cycle may be repeated as needed after a gap for a few seconds, in normal operation. The frequency of display, the interval between each flash, and the color of the LED may be adjusted accordingly, and the patterns may be customized differently for each platform, considering their complexity (e.g., fan tray size, accessibility, LED location, etc.) and the device heating curves during device module OIR.

Another example of a flash pattern that can be used to provide an indication of the determined OIR time having three digits may be as follows:If the OIR time is >4 minutes, provide a solid greenIf the OIR time is <4 minutes provide a solid green for 1 minute and thenflash green once for the first digit and then flash yellow the number of times needed to indicate the value of the first digit,flash green once for the second digit and then flash yellow the number of time needed to indicate the value of the second digit,flash green once for the third digit and then flash yellow the number of times needed to indicate the third digitsolid green for 1 minute, and repeat the flash sequence after every 1 minute of solid LED.

In addition, the solid green may instead be green or yellow/red based on the condition of the fans of the modular networking rack (to indicate all fans are working or display a fan fail condition)

A specific example may be as follows:If the OIR time is >4 minutes, provide a solid greenIf the OIR time is <4 minutes:provide a solid LED for 3 seconds to represent 0 oralternatively flash LED for 1 second on and off for 1 second to represent desired numbers, with a 5 second off time before displaying the next digitreturn to solid LED after the entire code is flashed.

For this example, flash patterns may be repeated every 60 seconds from start to start.

In some embodiments, different colors of LED may be used based on available OIR time. For example,<4 minutes to >90 seconds, flash code in green<90 seconds to >60 seconds, flash code in amber<60 seconds, continuously flash code in red

As noted above, a solid green LED may be green or yellow/red based on the condition of the fans in order to indicate all fans are working or that a fan fail condition exists.

In some embodiments, the indication of the OIR time may be provided for display to a user device, such as output via a custom smartphone application. In particular, the OIR time may be provided by connection with a network operations center (NOC) to use syslog information for real-time device temperature and to display OIR time limits. For example, by using a smartphone application, the user device may also monitor and control ambient temperature by adjusting central-office/data-center air conditioner units or other environmental control devices (or may be connected to the system controller which may be configured to adjust room temperature). The ambient temperature may thereby be lowered during OIR in order to gain additional time for module replacement. Bluetooth applications may also be used in which a Bluetooth sending unit may provide the OIR information via wireless connection to a user device (e.g., a smartphone or other handheld personal digital assistant) for immediate display.

In some embodiments, a liquid crystal sensor may be used to indicate the OIR time information to a user. In general, liquid crystal sensors use heat-sensitive (thermo-chromic) liquid crystals in a plastic strip which change color based on the ambient temperature. In particular, as shown inFIG. 5, device module500may include label550comprising liquid crystal thermometer560to display the ambient temperature around the modular networking rack containing the device module and, further, to mark the corresponding OIR time-limit. While this approach does not provide the OIR time as real time data but rather indicates the time-limit tested for a worst possible configuration for different operating ambient temperatures, label550positioned on a device module (such as a fan-tray) would provide a quick reference to an operator/user for the available time for device module replacement to be completed. In the case of a device module upgrade which could change the thermal profile of the system (e.g., high power, high airflow resistance, device thermal parameters), these time-limits would also need to be updated with a new label.

FIG. 6illustrates an example simplified procedure for displaying real-time module OIR time in accordance with one or more embodiments described herein. For example, a non-generic, specifically configured device (e.g., a system controller) may perform procedure600by executing stored instructions (e.g., an OIR determination and display process).

Procedure600may start at step605and continue to step610, where, as described in greater detail above, a device determines an OIR time for a particular device module of a plurality of device modules in a modular networking rack. The particular device module may be a line card, a fabric card, a route processor card, a power supply, or a fan tray. In some embodiments, the OIR time is the time needed to remove a fan tray from the modular networking rack. The device may determine the OIR time based on the operating temperature of the particular device, the air flow rate at the particular device module, a power dissipation rate of the particular device module, a maximum permissible temperature for the particular device module, a heating characteristic in no-airflow conditions for the particular device module, or combinations thereof.

In step615, as described in greater detail above, the device determines an OIR time for the plurality of device modules, based on the determined OIR time for the particular device module. In some embodiments, the device may compare the OIR of several particular device modules and determine, from the comparison, a minimum OIR is time for the plurality of device modules. Other factors may include device sensitivity, importance within the system, cost, and ease of replacement.

In step620, as described in greater detail above, the device provides an indication of the OIR time for the plurality of device modules for display to a user. The indication may be an electronic display, one or more LEDs, or a thermo-chromic strip. In some embodiments, one or more LED flash patterns may be used to display the determined OIR time. The flash patterns may include flash count/frequency, flash interval and/or duration, and flash color, which may be varied and used sequentially or in combination to indicate the OIR time value. In addition, the indication of the OIR time may be provided to a user device, such as a smartphone, which may, in some embodiments, be connected to room system devices such an air conditioner unit to modify the operating temperature and slow the temperature increase, particular for a fan tray OIR. Procedure600then ends at step625.

Thus, the techniques described herein provide a readily visible indication of OIR time for a networking rack containing a plurality of device modules, enabling a user to easily determine how much time is needed to replace a faulty or out of date device module, particularly a fan tray. In particular, an intelligent software process is described configured to estimate the real-time thermal status of the device module temperatures and available device module OIR time-limit and to provide a convenient user interface to display OIR information to the user. In this way, product reliability is improved and undesirable system shutdowns due to overheating or thermal runaway can be avoided. The techniques are independent of the system configuration and the operating ambient condition since it uses real-time device temperature information.

While there have been shown and described illustrative embodiments that provide for indicating module online insertion and removal (OIR) time, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the embodiments herein. For example, while exemplary embodiments relate to fan tray OIR, the techniques described herein may also be used for removal/replacement of other device modules, such as power supplies, where operating power levels and dissipation of power by device modules (time to minimum power) would be considered, in other embodiments.