Patent Description:
As such, organizations often seek out computing devices and network infrastructure that are individually and collectively configured to satisfy such regulations and standards. Specifically, organizations may use various techniques to enable the time of day (TOD) clocks at each device to be synchronized to the extent required in today's high-end computing architectures. For example, financial organizations may use one or more time servers that read a time from a reference clock and distribute this time to various devices within a network to improve a likelihood of each of these devices utilizing the correct time (e.g., such that each device then uses the received time as their respective TOD). Time servers frequently receive this reference clock from a global positioning system (GPS) signal.

Reference is made to <NPL>. This document describes a fundamental single point of failure in the PTPv2 protocol that affects its robustness to failure in specific error scenarios. The architecture design of electing a single unique time source to a PTP domain - the PTP GrandMaster - makes this protocol vulnerable to byzantine failures. Previous work has described this vulnerability from both a theoretical and practical point of view - and in particular how this affects the financial industry. This paper advances the discussion by contributing a description of the latest high-accuracy regulatory requirements on the financial industry, and by documenting new examples of failures in real-world customer-facing operations. It then describes an example of one of possible ways to increase PTP robustness while preserving its accuracy (using a multi-source NTP watchdog), and a laboratory test that shows how different protocol implementations are affected by this problem. In all, the current paper attempts to raise awareness of the robustness requirements within the financial industry today. As only PTP is accurate enough for both current and upcoming regulatory requirements, we hope that these issues are addressed in the forthcoming PTPv3 protocol, by adding multi-time source querying capabilities at the PTP end-slaves themselves.

<CIT> describes a method to provide a cyber attack resistant and fault tolerant precision clocking scheme for wide area critical infrastructure networks through what is called Distributed Time Source Validation, DTSV, is provided, which is a distributed algorithm and signaling mechanism for a network to detect a compromised time source or sources in a multiple master clock system. The method comprises providing a local clock signal, receiving in a node R1 an external clock signal from an external source, C or S1, estimating based on the local clock signal and the external clock signal timing parameters associated with the first node and the external source, comparing the timing parameters to detect any Mutual Clock Discrepancy (MCD) between the first node and the external source, and distributing any detected MCD in the network.

Preferably, there is provided a computer-implemented method comprising: comparing an average internal time of a plurality of data servers that each utilize a plurality of high-performance oscillators to maintain respective internal times as part of a network that utilizes precision time protocol (PTP) against other devices of the network; detecting (<NUM>), by analyzing the compared times, that a time maintained by another device of the network has drifted more than a threshold from the average internal time of all of the plurality of data servers; and executing (<NUM>, <NUM>) an ameliorative action in response to identifying that detecting that the time maintained by the another device of the network has drifted more than the threshold. More preferably, the another device includes another data server of the plurality of data servers; and the ameliorative action includes taking a high-performance oscillator of the another data server out of service in response to identifying that the high-performance oscillator is in error.

The drawings included in the present application are incorporated into, and form part of, the specification. They illustrate embodiments of the present disclosure and, along with the description, serve to explain the principles of the disclosure. The drawings are only illustrative of certain embodiments and do not limit the disclosure.

While the invention is amenable to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the invention.

Aspects of the present disclosure relate to managing clocks of computing devices of a network, while more particular aspects of the present disclosure relate to comparing an internal time of one or more data servers that use a plurality of high-performance oscillators to times of other devices of local or remote networks to identify, isolate, and execute ameliorative actions in response to identifying one or more devices of the network experiencing clock drift relative to these data servers. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.

As discussed in the background, many organizations are required (or otherwise inclined) to satisfy various regulations and standards in maintaining time synchronization across their computing devices (computing devices hereinafter referred to generically as "devices). This time synchronization regulation is often in relation to a universal standard (e.g., a standard that is relative to other organizations and/or relative to a regulatory body), such that what is important is not only whether devices have clocks that are synchronized relative to each other, also synchronized relative to these external entities. To meet such regulations, organizations typically use one or more servers which are dedicated to the task of gathering a time from a (presumably reliable) reference clock and then distributing this time to other devices of the network. Such servers that are assigned (if not dedicated) to gathering and distributing a time throughout a network are referred to herein as time servers. Time servers are distinct entities from "data servers" as discussed herein, where data servers are computing devices that are configured to store data and execute computing operations on behalf of the organization (and are not charged as being a primary time distribution device). Generally speaking, data servers may be understood to be mainframes or PC data servers of the organization.

Such conventional architectures are typically reliant upon various network components (e.g., a network switch) operating as expected. For example, if a network switch of an organization network starts malfunctioning (whether by delaying a time signal sent from the time server, changing a time signal sent from the time signal, or some other error as understood by one of ordinary skill in the art), then some computing devices that rely upon that network switch to receive clock information from the time server may drift from a true time as a result of these malfunctioning components.

Some conventional architectures attempt to solve this problem by making various efforts to ensure that all devices are generally synchronized with each other, such that none drift relative to each other. For example, some conventional architectures utilize precision time protocol (PTP) throughout a network, such as in conjunction with a synchronization program that uses a Yet Another Next Generation (YANG) model to ensure that all devices are synchronized. Additionally, or alternatively, some conventional architectures may utilize server time protocol (STP), which is a server-wide facility that presents a single view of time to relevant type-<NUM> hypervisors via STP messages transmitted over one or more physical data links between servers to improve a fidelity of messages transmitted throughout a network. Yet another example includes conventional architectures utilizing network time protocol (NTP) for clock synchronization. Using such techniques as this (whether alone or in conjunction with one or more of the procedures described above), conventional architectures may indeed be effective in ensuring that all devices are synchronized relative to each other.

However, conventional architectures may fail to identify a specific malfunctioning device that would cause single devices to drift. As would be understood by one of ordinary skill in the art, there are myriad reasons why it is beneficial to identify specific points of failure. For example, failing to identify a point of failure may make a conventional architecture susceptible to failure over time, such as if the number of malfunctioning devices multiply until a point where intra-network synchronization efforts are ineffective. For another example, failing to identify a point of failure may make it substantially more difficult for a conventional architecture to recover from an eventual failure (e.g., as it will not be known which components are required to be replaced/repaired). Specifically, following a failure, a conventional approach (where the failing devices is unknown) might include an extensive "trial and error" methodology where individual components are replaced and then the network is tested (where if that does not fix the problem, another component is replaced and the network is retested, etc.).

Beyond this, even if such conventional intra-network synchronization efforts succeed in getting all devices of a network to be synchronized relative to each other, they might not ensure that these devices are synchronized as required relative to external clocks. For example, it may be difficult or impossible for conventional synchronization efforts of conventional architectures to detect if the time server itself is malfunctioning or is receiving a corrupted time source. For example, if a malicious actor spoofs the reference clock signal used by a time server (e.g., such as the GPS signal, or even a PTP signal for some conventional architectures), conventional synchronization efforts may be technically incapable of detecting that the time received and distributed by the time server is not synchronized relative to external (e.g., true/actual) times, such as UTC.

In some situations, a network may attempt to solve this by including a single high-quality oscillator that is configured to maintain an internal time within at least one computing device of the network. However, while such a device may be configured to determine that something is wrong when its internal time did not match an external time, this device would be unable to determine if the failure is with itself or with the external device.

Aspects of this disclosure may solve or otherwise address these technical problems of conventional computing architectures. For example, the above technical problems are solved using data servers that includes multiple high-performance oscillators (e.g., where the oscillator is high-performance as a result of the oscillator being specified upon its own construction to approximately ±<NUM> parts per million) that are configured to maintain an internal time of the data server. Such data servers may further include software that works in conjunction with the plurality of high-performance oscillators to maintain the internal time (e.g., to drift no more than <NUM> milliseconds a day). Aspects of the disclosure relate to comparing an internal time of day (TOD) time of one or more such data servers to internal TOD times of various devices to detect a device drifting (where drifting, as used herein, relates to a computing device getting ahead or behind a desired time by a non-nominal amount that exceeds a threshold and therein warrants correction), and therein executing an ameliorative action in response to such a detection. A computing device that includes a processing unit executing instructions stored on a memory may provide this functionality, this computing device referred to herein as a controller. By comparing various internal times of various devices with one or more data servers that utilizes multiple high-performance oscillators as described herein, the controller may be configured to detect whether any device within a computing environment device is drifting, and moreover identify whether the drift was caused by a failing device or by an error/attack relating to the incoming reference clock.

For example, <FIG> depicts environment <NUM> in which controller <NUM> monitors and manages internal time drift of data servers 120A, time servers 120B, and other devices 120C (where data servers 120A, time servers 120B, and other devices 120C are collectively referred to as "devices <NUM>" herein). Controller <NUM> may include a computing device, such as computing system <NUM> of <FIG> that includes a processor communicatively coupled to a memory that includes instructions that, when executed by the processor, causes controller <NUM> to execute one or more operations described below. For example, controller <NUM> may monitor and manage time drift of any devices <NUM> on network <NUM>. As discussed herein, all devices <NUM> on network <NUM> are to be synchronized to a true time such as the UTC (which may otherwise be phrase as the entirety of network <NUM> is to be synchronized to an external time).

Each data server 120A includes a plurality of high-performance oscillators <NUM> configured to maintain an internal time of the respective data server 120A as described herein. For example, high-performance oscillators <NUM> may be specified to be accurate at a range between ± <NUM> to <NUM> parts per million. Data servers 120A have at least two and as many as eight high-performance oscillators <NUM>, though data servers 120A are predominantly discussed as having four high-performance oscillators <NUM>. Each oscillator <NUM> may be configured to individually keep track of an internal time of data server 120A, where an eventual TOD for the respective data server 120A is an average time of each of these oscillators <NUM> (e.g., a mean, median, mode, of these respective times). In this way, each additional high-performance oscillator <NUM> may provide an additional "vote" in determining what the correct time is, such that the more oscillators <NUM> exist, the more robust the time synchronization efforts are (but also the more expensive each data server 120A is).

In some examples data servers 120A further utilize software to stabilize the internal time. This may include reducing drift by approximately <NUM> or <NUM> orders of magnitude better than what is enabled by high-performance oscillators <NUM> alone, such that a daily drift would be no more than a range of <NUM> milliseconds to <NUM> milliseconds per day.

As depicted, numerous time servers 120B are on network <NUM>, but in other examples network <NUM> may be served by a single time server 120B. Time servers 120B are configured to receive or read a time from a reference clock, and then distribute this time to some or all devices <NUM> of network <NUM>. Time servers 120B are computing devices with components that are similar to computing system <NUM> of <FIG> (e.g., such that time servers 120B includes interface <NUM>, processor <NUM>, and memory <NUM> as discussed in relation to <FIG> in some capacity). Time servers <NUM> may be understood to not necessarily include high performance oscillators <NUM>.

Comparatively, data servers 120A are not used within network <NUM> for a default task of receiving and distributing a reference clock signal throughout network <NUM> (e.g., such that upon initializing network <NUM>, data servers 120A are not assigned an initial task of gathering a time from a reference clock and distributing a clock signal to device <NUM> of network <NUM>). Rather, data servers 120A may be understood to be mainframe computing devices or PC data servers. As such, data servers 120A may be understood to be used for primary computing operations for an organization that is utilizing network <NUM> (e.g., such that data servers 120A are part of the central data repository for the organization).

Other devices 120C include computing devices of network <NUM> that are to be synchronized along with data servers 120A within network <NUM> but are not time servers 120B and are more capable of drifting than data servers 120A. For example, other devices 120C may include servers that do not have any high-performance oscillators <NUM>, or perhaps have a single high-performance oscillator <NUM>, or are otherwise not configured to be as drift-resistant as data servers 120A. In some examples, other devices 120C are mainframe computing devices or PC data servers. Both data servers 120A and other devices 120C are computing devices with components that are similar to computing system <NUM> of <FIG> (e.g., such that both include interface <NUM>, processor <NUM>, and memory <NUM> in some capacity as discussed in relation to <FIG>).

In some examples, controller <NUM> is separate from devices <NUM> as depicted in <FIG>, such that controller <NUM> manages time synchronization of network <NUM> as part of a computing device that is physically discrete relative to devices <NUM>. In other examples (not depicted), controller <NUM> may be integrated into one or many of devices <NUM> (e.g., perhaps as a distributed system). For example, controller <NUM> may be integrated into one data server 120A, and/or controller <NUM> may be integrated as individual instances into each or many of data servers 120A.

As discussed herein, time servers 120B send a reference clock signal to data servers 120A and other devices 120C. Time servers 120B use a plurality of switches <NUM> to send the reference clock signal through network <NUM> to data servers 120A and other devices 120C. Switches <NUM> are network switches that connect devices <NUM> via techniques such as packet switching on one or more layers of the open systems interconnection (OSI) model.

Network <NUM> may include one or more computer communication networks. An example network <NUM> can include the Internet, a local area network (LAN), a wide area network (WAN), a wireless network such as a wireless LAN (WLAN), or the like. Network <NUM> may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. That said, connections of network <NUM> that are between time servers 120B and devices <NUM> may only utilize such connections as are capable of the high-speed data transmission required for data synchronization as described herein. For example, each of devices <NUM> and switches <NUM> may be connected to respective time servers 120B that serve these devices <NUM> over a LAN. A network adapter card or network interface in each computing/processing device (e.g., controller <NUM>, data server 120A, time server 120B, other devices 120C) may receive messages and/or instructions from and/or through network <NUM> and forward the messages and/or instructions for storage or execution or the like to a respective memory or processor of the respective computing/processing device.

Though network <NUM> is depicted as a single entity in <FIG> for purposes of illustration, in other examples network <NUM> may include a plurality of private and/or public networks over which controller <NUM> may manage time as described herein. For example, in some situations network <NUM> may include two clustered subnetworks in which devices <NUM> are connected via respective LANs, and additionally connected via a WAN or the like even as these two clustered subnetworks are geographically dispersed. Specifically, the two clustered subnetwork may be located, e.g., in different buildings, different cities, or otherwise on the realm of <NUM>,<NUM> kilometers away. In this example, each of the two geographically dispersed clustered subnetworks includes at least one time server 120B and at least one data server 120A, and controller <NUM> (whether one controller <NUM> or distinct instances of controller <NUM>) manages time synchronization and time drift of devices <NUM> in the two geographically dispersed clustered subnetworks.

Controller <NUM> detects that a time maintained by at least one device <NUM> of network <NUM> is more than a threshold away from a time maintained by at least one data server 120A. For example, a threshold may be <NUM> microsecond, <NUM> microseconds, or <NUM> microseconds, and controller <NUM> may detect that a time of a TOD clock of one of devices <NUM> is <NUM> microseconds, <NUM> microseconds, or <NUM> microseconds (respectively) away from a time maintained by a single data server 120A, and therefore exceeds the respective threshold. While these specific threshold numbers are provided for purposes of discussion, one of ordinary skill in the art would understand that such numbers are heavily dependent upon the regulations related to the organization of network <NUM> and the specifications/capabilities of devices <NUM> of network <NUM> (e.g., such that devices <NUM> that are capable of tighter tolerances might have smaller thresholds, and/or organizations that are held to "lower" regulations might have notably larger thresholds). Therefore, one of ordinary skill in the art would understand that any user-defined threshold that identifies a drift that is both larger than a time synchronization that is capable of being maintained by devices <NUM> and also approaching (or potentially failing) the allowable limit of drift as defined by various regulations and/or standards applicable to the organization is consistent with this disclosure.

Controller <NUM> executes an ameliorative action in response to detecting this drift by more than the threshold. An ameliorative action may include invoking the best master clock algorithm, invoking STP links, changing a clock signal from being propagated throughout network <NUM> by time server 120B being propagated throughout network <NUM> by one or more data servers 120A to devices <NUM>, notifying an admin of the time drift (e.g., including identifying which device <NUM> has drifted by what amount), taking one or more malfunctioning oscillators <NUM> out of service, or the like.

In some examples, controller <NUM> may compare a time of devices <NUM> against a single data server 120A. For example, a local network <NUM> could include one single data server 120A, and controller <NUM> may execute an ameliorative action as discussed herein in response to detecting that any of devices <NUM> were more than a threshold amount of time away from an internal time of this single data server 120A. For example, controller <NUM> could detect that time server 120B is malfunctioning (or has received a bad reference signal) as a result of detecting that the time of time server 120B is different than one single data server 120A.

In other examples, a single network <NUM> on one LAN (e.g., within a single room, or a single building) may include numerous data servers 120A, and controller <NUM> may compare a time of individual devices <NUM> against the average time of multiple data servers 120A. Controller <NUM> may calculate an average time of multiple data servers 120A via any number of statistical methods, such as by calculating a mean, median, mode, or some other statistical method. For example, controller <NUM> may compare a time of one time server 120B of a LAN of network <NUM> against some or all data servers 120A of that network <NUM> LAN.

Controller <NUM> may compare an internal time of devices <NUM> to the time of data servers 120A on a set schedule, and/or in response to a condition being met. For example, controller <NUM> may compare an internal time of each of devices <NUM> against the average time of data servers 120A once every <NUM> minutes, once every hour, once every <NUM> hours, once every day, once every few days, or the like. Controller <NUM> may compare a time of devices <NUM> against the time of data servers 120A more frequently to catch a potential drift sooner, whereas controller <NUM> may compare less frequently to use less computing resources. In some examples, controller <NUM> may be configured to compare times of devices <NUM> against times of data servers 120A in response to a resource utilization falling below a threshold (e.g., in response to a processing, memory, and or bandwidth utilization rate of network <NUM> falling below some percentage, indicating that there is surplus computing resources for use). Additionally, or alternatively, controller <NUM> may be configured to compare a time of devices <NUM> to a time of data servers 120A in response to something being detected that indicates drifting (e.g., an error, alert, or condition that is correlated to one or more devices <NUM> drifting).

Controller <NUM> may execute ameliorative actions autonomously. Specifically, controller <NUM> may execute ameliorative actions as discussed herein without intervention from a human. Beyond this, controller <NUM> may execute ameliorative actions nearly immediately upon detecting that a time of one or more of devices <NUM> is drifting, such as within a millisecond or a second of such detection. By being configured to autonomously and nearly immediately execute an ameliorative action in response to detecting any devices <NUM> drifting, aspects of this controller <NUM> may improve the likelihood of devices <NUM> utilizing the correct time (and therein reduce the likelihood that an organization that is using these devices <NUM> will have to pay a fine or the like as a result of any devices <NUM> not utilizing the correct time for an extended period of time).

As discussed herein, in some examples controller <NUM> detects that a time as maintained and/or received by time server 120B of network <NUM> is more than a threshold away from the average time of a plurality of data servers <NUM>. In response to such a detection, controller <NUM> executes an ameliorative action. For example, controller <NUM> may notify an admin and also cause devices <NUM> of environment <NUM> to receive a clock signal that is the average time of the plurality of data servers <NUM> (rather than the clock signal from the time server 120B that is drifting). In this way, in response to detecting that a time of time server 120B is drifting relative to the average time of a plurality of data servers 120A, controller causes devices <NUM> to utilize the average internal time of the plurality of data servers 120A.

In certain examples, controller <NUM> may compare devices <NUM> of one geographic location against data servers 120A of a different geographic location. For example, controller <NUM> may compare data servers 120A of one geographic location against data servers 120A of a second geographic location. Alternatively, or additionally, controller <NUM> may compare how devices <NUM> of a first location are drifting relative to data servers 120A of that first location, and then compare that drift against how devices <NUM> of a second location are drifting relative to data servers of that second location. In this way, aspects of this disclosure may be configured to enable tight time synchronization for widely dispersed networks, such as a graphically dispersed parallel sysplex (GDPS).

In other examples, controller <NUM> may compare a time of a first time server 120B at one location against both a time server 120B and data servers 120A of a second location. Specifically, controller <NUM> may detect that time server 120B at a first location has an incorrect time, and may compare this incorrect time against a time of a time server 120B at a second location that is geographically dispersed from the first location. Where controller <NUM> detects that these two time servers 120B both have incorrect times that are within a threshold of each other (e.g., within <NUM> or <NUM> microseconds of each other), controller <NUM> may conclude that the problem is likely with a time source rather than the time servers 120B themselves. In this way, aspects of the disclosure may be configured to determine that, e.g., a malicious third party appears to be tampering with a time source, such as a GPS signal (e.g., via GPS spoofing). In response to such a determination, controller <NUM> may cause devices <NUM> of both locations to instead use times of data servers 120A rather than time servers 120B, at least until an admin can verify conditions of the two locations.

Controller <NUM> is configured to detect if one data server 120A is drifting relative to other data servers 120A. Where controller <NUM> detects that one data server 120A is drifting relative to other data servers 120A, controller <NUM> may analyze a performance of each oscillator <NUM> of this drifting data server 120A. In many cases, controller <NUM> will identify at least one oscillator <NUM> of the drifting data server 120A that is in error and is therein causing this data server 120A to drift. In response to detecting one or more oscillators <NUM> that are in error, controller <NUM> may execute an autonomous action of taking these erring oscillators <NUM> out of service within the drifting data server 120A. Controller <NUM> may further notify an admin and/or request a replacement oscillator <NUM> for the (previously) drifting data server 120A.

Additionally, or alternatively, controller <NUM> may take data server 120A out of service in response to detecting that data server 120A was drifting (in response to erring oscillator <NUM>). In other examples, controller <NUM> may treat this previously drifting data server 120A as a new other device 120C rather than a data server 120A in response to detecting that this previously drifting data server <NUM> was drifting (and/or in response to taking one oscillator <NUM> out of service). Put differently, controller <NUM> may not compare other devices <NUM> against this previously drifting data server 120A until this previously drifting data server 120A is fully serviced and repaired, such that this previously drifting data server 120A is confirmed to have a performance along the lines of other data servers 120A (e.g., four working oscillators <NUM> that each are specified to ±<NUM> ppm).

Controller <NUM> may be configured to detect when one other device 120C is drifting. In response to controller <NUM> detecting that one other devices 120C is drifting, controller <NUM> may check whether or not any more other devices 120C are drifting. If more other devices 120C are drifting, controller <NUM> may compare the drift between these other devices 120C to see if they are similar. Where controller <NUM> determines that more than one other devices 120C are drifting a similar amount, controller <NUM> may identify commonalities of a route through which these other devices 120C received a clock signal from time server 120B. For example, controller <NUM> may determine that within network <NUM> that includes forty other devices 120C, that eight other devices 120C are experiencing substantially identical drifts, and moreover that all of these eight other devices 120C share a common switch <NUM>. In response to this determination that a common switch <NUM> is involved in all drifting other devices 120C, controller <NUM> may execute an ameliorative action of rerouting the clock signal to the previously drifting other devices 120C (e.g., through a new route that avoids the problematic switch <NUM>). Controller <NUM> may also notify an admin of the seemingly malfunctioning switch <NUM>.

In other examples, upon analyzing all other devices 120C (in response to detecting that one other device 120C is drifting), controller <NUM> may determine that no additional other devices 120C is drifting (or drifting in a similar manner) beyond this one other device 120C. In response to such a determination, controller <NUM> may execute an ameliorative action that includes one or more of reporting this problem to an admin, invoking the best master clock algorithm, switching the drifting other device 120C to receive the time from a neighboring device <NUM>, or the like.

As described above, controller <NUM> may be part of a computing device that includes a processor configured to execute instructions stored on a memory to execute the techniques described herein. For example,.

<FIG> is a conceptual box diagram of such computing system <NUM> of controller <NUM>. While controller <NUM> is depicted as a single entity (e.g., within a single housing) for the purposes of illustration, in other examples, controller <NUM> may include two or more discrete physical systems (e.g., within two or more discrete housings). Controller <NUM> may include interfaces <NUM>, processor <NUM>, and memory <NUM>. Controller <NUM> may include any number or amount of interface(s) <NUM>, processor(s) <NUM>, and/or memory(s) <NUM>.

Controller <NUM> may include components that enable controller <NUM> to communicate with (e.g., send data to and receive and utilize data transmitted by) devices that are external to controller <NUM>. For example, controller <NUM> may include interface <NUM> that is configured to enable controller <NUM> and components within controller <NUM> (e.g., such as processor <NUM>) to communicate with entities external to controller <NUM>. Specifically, interface <NUM> may be configured to enable components of controller <NUM> to interact with devices <NUM>, switches <NUM>, or the like. Interface <NUM> may include one or more network interface cards, such as Ethernet cards and/or any other types of interface devices that can send and receive information. Various numbers of interfaces may be used to perform the described functions according to particular needs.

As discussed herein, controller <NUM> may be configured to manage time synchronization within a computing network. Controller <NUM> may utilize processor <NUM> to thusly manage time. Processor <NUM> may include, for example, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or equivalent discrete or integrated logic circuits. Two or more of processor <NUM> may be configured to work together to identify whether or not any devices <NUM> are drifting and execute ameliorative actions accordingly.

Processor <NUM> may manage time of devices <NUM> in environment <NUM> according to instructions <NUM> stored on memory <NUM> of controller <NUM>. Memory <NUM> may include a computer-readable storage medium or computer-readable storage device. In some examples, memory <NUM> includes one or more of a short-term memory or a long-term memory. Memory <NUM> may include, for example, random access memories (RAM), dynamic random-access memories (DRAM), static random-access memories (SRAM), magnetic hard discs, optical discs, floppy discs, flash memories, forms of electrically programmable memories (EPROM), electrically erasable and programmable memories (EEPROM), or the like.

In addition to instructions <NUM>, in some examples gathered or predetermined data or techniques or the like as used by processor <NUM> to manage time drift and synchronization as described herein is stored within memory <NUM>. For example, memory may also include time data <NUM>, which may include various thresholds and schedules at which controller <NUM> is to monitor internal time of devices <NUM>. Memory <NUM> may also include data server data <NUM>, time server data <NUM>, and other device data <NUM>. Data server data <NUM> may include historical and/or current time data for data servers 120A, while time server data <NUM> may include historical and/or current time data for time servers 120B, and other device data <NUM> includes historical and/or current time data for other devices 120C.

Memory <NUM> may further include machine learning techniques <NUM> that controller <NUM> may use to improve a process of managing time synchronization and drifting as discussed herein over time. Machine learning techniques <NUM> can comprise algorithms or models that are generated by performing supervised, unsupervised, or semi-supervised training on a dataset, and subsequently applying the generated algorithm or model to monitor time synchronization or drift as described herein. For example, using machine learning techniques <NUM>, controller <NUM> may determine that certain drift thresholds for certain types of devices <NUM> are more indicative of drift, and/or are more likely to result in a problematic drift before a next schedule scan. For another example, controller <NUM> may use machine learning techniques <NUM> to determine that certain types of ameliorative actions are better or worse at reducing drift over time. Controller <NUM> may reinforce rules over time based on whether an ability to reduce time drift improves or declines based on rule updates. For example, controller <NUM> may track whether or not any an amount of drifts that required ameliorative actions are increasing or decreasing and either change or stabilize future actions accordingly.

Machine learning techniques <NUM> can include, but are not limited to, decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity/metric training, sparse dictionary learning, genetic algorithms, rule-based learning, and/or other machine learning techniques.

For example, machine learning techniques <NUM> can utilize one or more of the following example techniques: K-nearest neighbor (KNN), learning vector quantization (LVQ), self-organizing map (SOM), logistic regression, ordinary least squares regression (OLSR), linear regression, stepwise regression, multivariate adaptive regression spline (MARS), ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS), probabilistic classifier, naïve Bayes classifier, binary classifier, linear classifier, hierarchical classifier, canonical correlation analysis (CCA), factor analysis, independent component analysis (ICA), linear discriminant analysis (LDA), multidimensional scaling (MDS), non-negative metric factorization (NMF), partial least squares regression (PLSR), principal component analysis (PCA), principal component regression (PCR), Sammon mapping, t-distributed stochastic neighbor embedding (t-SNE), bootstrap aggregating, ensemble averaging, gradient boosted decision tree (GBRT), gradient boosting machine (GBM), inductive bias algorithms, Q-learning, state-action-reward-state-action (SARSA), temporal difference (TD) learning, apriori algorithms, equivalence class transformation (ECLAT) algorithms, Gaussian process regression, gene expression programming, group method of data handling (GMDH), inductive logic programming, instance-based learning, logistic model trees, information fuzzy networks (IFN), hidden Markov models, Gaussian naïve Bayes, multinomial naïve Bayes, averaged one-dependence estimators (AODE), Bayesian network (BN), classification and regression tree (CART), chi-squared automatic interaction detection (CHAID), expectation-maximization algorithm, feedforward neural networks, logic learning machine, self-organizing map, single-linkage clustering, fuzzy clustering, hierarchical clustering, Boltzmann machines, convolutional neural networks, recurrent neural networks, hierarchical temporal memory (HTM), and/or other machine learning algorithms.

Using these components, controller <NUM> may manage time synchronization and drift as discussed herein. In some examples, controller <NUM> manages time synchronization of devices <NUM> according to flowchart <NUM> depicted in <FIG>. Flowchart <NUM> of <FIG> is discussed with relation to <FIG> for purposes of illustration, though it is to be understood that other systems and message may be used to execute flowchart <NUM> of <FIG> in other examples. Further, in some examples controller <NUM> executes a different method than flowchart <NUM> of <FIG>, or controller <NUM> executes a similar method with more or less steps in a different order, or the like.

Flowchart <NUM> starts with controller <NUM> monitoring an internal time (e.g., a TOD) of devices <NUM> of network <NUM> (<NUM>). Controller <NUM> may determine whether or not data servers 120A are in agreement with each other, such that all data servers 120A are within a tight tolerance/threshold of each other (<NUM>). If controller <NUM> determines that any data servers 120A are outside of a threshold (no branch from <NUM>), controller <NUM> may attempt to identify whether any oscillators <NUM> are in error (<NUM>). Controller <NUM> may then execute an ameliorative action (<NUM>), whether taking an erring oscillator <NUM> out of service, taking the erring data server 120A out of service, requesting a replacement oscillator <NUM>, notifying an admin, treating the erring data server 120A as an other device 120C, or the like.

If controller <NUM> determines that all data servers 120A are in agreement (yes branch from <NUM>), controller <NUM> determines whether or not other devices 120C are in agreement within a threshold (<NUM>). If other devices 120C are in agreement (yes branch from <NUM>), controller <NUM> continues monitoring time of devices <NUM> (e.g., at a next scheduled time). If other devices 120C are not in agreement (no branch from <NUM>), controller <NUM> may compare the drift experienced locally against a drift experienced by geographically dispersed devices <NUM> (e.g., other devices <NUM> of a shared WAN) (<NUM>).

Controller <NUM> may verify whether or not geographically dispersed devices <NUM> are experiencing a similar drift (<NUM>). For example, controller <NUM> may determine that all other devices 120C of a first geographic location served by a first time server 120B are experiencing a drift of a first magnitude, and all other devices 120C of a second geographic location serviced by a second time server 120B are experiencing a drift of a second magnitude, where the first drift and the second drift are substantially similar (yes branch from <NUM>).

One of ordinary skill in the art would understand that a drift would be understood to be substantially similar if the two drifts are so close such that it is unlikely to be a coincidence, and that rather it is more likely that the two drifts are the result of both receiving the same (potentially malicious) incorrect reference clock. The exact value that would merit such a determination might change depend upon the situation, such as the accuracy of the components and/or the precision of a potential attack, though an example threshold to be identified as substantially similar might be within <NUM> seconds of each other. In response to this determination, controller <NUM> may execute an ameliorative action for geographically dispersed drift (<NUM>). This may include notifying an admin that a time source appears to have been spoofed, changing a time source for a given network <NUM> to be a(n average) time of data servers 120A rather than the gathered reference time of time servers 120B, or the like. After executing this ameliorative action, controller <NUM> may return to monitoring devices <NUM> (<NUM>).

If controller <NUM> determines that a geographically dispersed location is not experiencing a similar drift (no branch from <NUM>), controller <NUM> may track the clock signal to identify a local component error (<NUM>). A component may include switches <NUM>, other devices 120C, and/or time servers 120B. For example, if numerous other devices 120C are all drifting and all share a respective switch <NUM>, controller <NUM> may identify that the respective switch <NUM> is in error and may execute an ameliorative action (<NUM>) of routing the clock signal to these other devices 120C without going through this respective switch <NUM>. If controller <NUM> determines that numerous other devices 120C that are serviced by a single time server 120B are drifting, then controller <NUM> may execute an ameliorative action (<NUM>) of causing these other devices 120C to instead receive a clock signal from data servers 120A as discussed herein (or cause these other devices 120C to be serviced by another time server 120B of network <NUM>, as applicable). Alternatively, if controller <NUM> determines that a single other device 120C was drifting, then controller <NUM> may execute an ameliorative action (<NUM>) of executing STP and/or causing this single other device 120C to receive time from a nearby device <NUM>.

The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. The terminology used herein was chosen to explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-situation data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the "C" programming language or similar programming languages.

Claim 1:
A computer-implemented method comprising:
comparing an internal time of a data server on a network against respective times of each of plurality of devices on the network, wherein the data server comprises a plurality of high-performance oscillators to maintain the internal time of the data server on the network, wherein each high-performance oscillator in the plurality of high-performance oscillators keeps track of the internal time of the data server individually, and wherein the internal time of the data server is an average time obtained from the plurality of high-performance oscillators;
detecting (<NUM>), by analyzing the compared times, that a time maintained by another device of the network has drifted more than a threshold; and
executing (<NUM>, <NUM>) an ameliorative action in response to detecting that the time maintained by the another device has drifted more than the threshold.