DUAL CHANNEL NETWORK FOR MULTIVARIATE TIME SERIES RETRIEVAL WITH STATIC STATUSES

A computer implemented method is provided. The method includes jointly encoding, by a dual-channel feature extractor, a current time series segment with corresponding static statuses into a compact feature. The method further includes converting, by a binary code extractor, the compact feature into a binary code. The method also includes computing distances between the binary code and all binary codes stored in a binary code database. The method additionally includes retrieving the top relevant multivariate time series segments based on the distances.

BACKGROUND

Technical Field

The present invention generally relates to time series processing and more particularly to a dual channel network for multivariate time series retrieval with static statuses.

Description of the Related Art

Multivariate time series retrieval is the task of finding the most relevant multivariate time series segments from a huge amount of historical data by querying with a current observation. A feasible way to perform multivariate time series retrieval is to obtain compact representation of the historical data with binary codes that preserve relative similarity relation in raw input space. However, the system status is not always determined only by time series that describes the dynamic system behavior but sometimes also by static status of the system.

SUMMARY

According to aspects of the present invention, a computer implemented method is provided. The method includes jointly encoding, by a dual-channel feature extractor, a current time series segment with corresponding static statuses into a compact feature. The method further includes converting, by a binary code extractor, the compact feature into a binary code. The method also includes computing distances between the binary code and all binary codes stored in a binary code database. The method additionally includes retrieving the top relevant multivariate time series segments based on the distances.

According to other aspects of the present invention, a computer program product is provided. The computer program product includes a non-transitory computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes jointly encoding, by a dual-channel feature extractor implemented by one or more hardware processors of the computer, a current time series segment with corresponding static statuses into a compact feature. The method further includes converting, by a binary code extractor implemented by the one or more hardware processors, the compact feature into a binary code. The method also includes computing, by the one or more hardware processors, distances between the binary code and all binary codes stored in a binary code database. The method additionally includes retrieving, by the one or more hardware processors, the top relevant multivariate time series segments based on the distances.

According to still other aspects of the present invention, a computer processing system is provided. The computer processing system includes a memory device for storing program code. The computer processing system further includes one or more hardware processors for running the program code to jointly encode, by a dual-channel feature extractor implemented by the one or more hardware processors, a current time series segment with corresponding static statuses into a compact feature. The one or more hardware processors further run the program code to convert, by a binary code extractor implemented by the one or more hardware processors, the compact feature into a binary code. The one or more hardware processors also run the program code to compute distances between the binary code and all binary codes stored in a binary code database. The one or more hardware processors additionally run the program code to retrieve the top relevant multivariate time series segments based on the distances.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Embodiments of the present invention are directed to a dual channel network for multivariate time series retrieval with static statuses.

One or more embodiments of the present invention provide an end-to-end neural network model architecture that also considers static statuses as well as time series inputs for more accurate multivariate time series retrieval.

As mentioned above, conventional state-of-the-art techniques for time series retrieval assume the system status is determined only by dynamic behavior represented as time series. However, in practice, the system status is also affected by static observations such as operational mode, day of the week. A recurrent neural network based time series encoder, which is typically used for time series retrieval, cannot handle such static inputs since it assumes temporal dependency among time stamps.

One or more embodiments of the present invention address time series retrieval tasks for systems whose status is determined not only by dynamic behavior but also by static profiles. Embodiments of the present invention incorporate a multi-layer perceptron based static encoder as well as a recurrent neural network based temporal encoder. These encoders are jointly trained in end-to-end manner based on metric learning loss such as triplet loss.

Hence, in an embodiment, at least one of the following two inventive features can be involved: (1) Multiple Layer Perceptron (MLP) based static encoder is jointly trained with a Recurrent Neural Network (RNN) based time series encoder; (2) metric learning loss enables us to train binary codes that preserve relative similarity between input raw time series as well as static statuses.

The task of multivariate time series retrieval can be applied to many tasks in complex systems including system status identification, fault detection and fault prediction to name a few exemplary tasks to which embodiments of the present invention can be applied.

As used herein, “static statuses” refer to system statuses represented that do not change within the same consecutive time points as time series segments. The difference between static statuses and time series is the frequency of change. While time series changes every moment, static statuses changes only at certain time intervals.

FIG.1is a block diagram showing an exemplary computing device100, in accordance with an embodiment of the present invention. The computing device100is configured to perform multivariate time series retrieval with static statuses for a dual channel network.

The computing device100may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a server, a rack based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and/or a consumer electronic device. Additionally or alternatively, the computing device100may be embodied as a one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically disaggregated computing device. As shown inFIG.1, the computing device100illustratively includes the processor110, an input/output subsystem120, a memory130, a data storage device140, and a communication subsystem150, and/or other components and devices commonly found in a server or similar computing device. Of course, the computing device100may include other or additional components, such as those commonly found in a server computer (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory130, or portions thereof, may be incorporated in the processor110in some embodiments.

The processor110may be embodied as any type of processor capable of performing the functions described herein. The processor110may be embodied as a single processor, multiple processors, a Central Processing Unit(s) (CPU(s)), a Graphics Processing Unit(s) (GPU(s)), a single or multi-core processor(s), a digital signal processor(s), a microcontroller(s), or other processor(s) or processing/controlling circuit(s).

The memory130may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory130may store various data and software used during operation of the computing device100, such as operating systems, applications, programs, libraries, and drivers. The memory130is communicatively coupled to the processor110via the I/O subsystem120, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor110the memory130, and other components of the computing device100. For example, the I/O subsystem120may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, platform controller hubs, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem120may form a portion of a system-on-a-chip (SOC) and be incorporated, along with the processor110, the memory130, and other components of the computing device100, on a single integrated circuit chip.

The data storage device140may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. The data storage device140can store program code for multivariate time series retrieval with static statuses for a dual channel network. The communication subsystem150of the computing device100may be embodied as any network interface controller or other communication circuit, device, or collection thereof, capable of enabling communications between the computing device100and other remote devices over a network. The communication subsystem150may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

As shown, the computing device100may also include one or more peripheral devices160. The peripheral devices160may include any number of additional input/output devices, interface devices, and/or other peripheral devices. For example, in some embodiments, the peripheral devices160may include a display, touch screen, graphics circuitry, keyboard, mouse, speaker system, microphone, network interface, and/or other input/output devices, interface devices, and/or peripheral devices. The peripheral devices can also include motor vehicle systems including steering, braking, accelerating, lighting, stability, and so forth, as described herein.

In other embodiments, the hardware processor subsystem can include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry can include one or more application-specific integrated circuits (ASICs), FPGAs, and/or PLAs.

FIG.2is a block diagram showing an exemplary dual channel feature extractor200, in accordance with an embodiment of the present invention.FIG.3is a block diagram showing the dual channel feature extractor220ofFIG.2, in accordance with an embodiment of the present invention.

The dual channel feature extractor200includes a data preprocessor210, a dual channel feature extractor220, a binary code extractor230, and a binary code database240. InFIGS.2and3, “f” denotes a feature vector, “b” denotes a binary code, and “h” denotes a hidden layer output.

The data preprocessor210preprocesses raw data to extract static status as well as time series for each time step. Specifically, for a consecutive T number of time points, a time series segment Xt=[xt,xt+1, . . . xt+T−1] ∈d1×Tare extracted from the raw time series, and static statuses st∈d2are extracted from the raw static profile, at each time point t=0,1, . . . , L, where L is the length of whole raw time series.

The dual channel feature extractor220jointly encodes the static status and the time series, which are preprocessed by data preprocessor210into compact representations. The dual channel feature extractor220includes a network architecture based on a Multi-Layer Perceptron (MLP)221and a Recurrent Neural Network (RNN)222. The dual channel feature extractor220further includes a binary prediction layer232which reshapes the size of features from the concatenation of the outputs of the MLP221and RNN222and predicts binary codes.

The binary code extractor230converts the compact representations encoded by the dual channel feature extractor220into binary codes by checking the sign of all entries in the feature vector.

The binary code database240stores all the historical binary codes, which are extracted by the binary code extractor230.

In an embodiment, the present invention has three stages: a training stage; a hashing stage; and a retrieval stage. In the training stage, network parameters of the dual-channel feature extractor220are trained on all historical time series and their static statuses. After the training stage, all historical time series and static statuses are encoded into binary codes based on the dual-channel feature extractor220trained in the training stage, and then stored in the binary code database240. In the retrieval stage, for each incoming time series and static statuses, the most relevant time series as well as static statuses are retrieved by comparing extracted binary codes to all historical ones in the binary code database240.

FIG.4is a flow diagram showing an exemplary training method400, in accordance with an embodiment of the present invention.

At block410, extract, by the data preprocessor210, multivariate time series segments and corresponding static statuses from historical data.

At block420, jointly encode, by the dual-channel feature extractor220, the multivariate time series segments and the corresponding static statuses into compact representations. It is to be appreciated that the terms “compact features” and “compact representations” are used interchangeably herein. A compact feature is a feature having a certain dimension, which is much smaller than the multiplication of the original dimension and the length of time series segments.

At block430, evaluate the encoded representation by supervised metric learning loss.

At block440, update network parameters of the dual-channel feature extractor220.

At block450, determine if the stopping condition is satisfied. If so, then terminate the method. Otherwise, return to step420.

In the training stage, further regarding block410, multivariate time series segments (a slice of multivariate time series that lasts for a certain time steps) and corresponding static status are extracted. Multivariate time series segments are extracted from entire multivariate time series by a sliding window.

Further regarding block420, time series segments with static statuses are encoded by the dual-channel feature extractor220into compact representations.

Specifically, a time series segment Xt=[xt,xt+1, . . . xt+T−1] ∈d1×Twith static statuses st∈d2at time t are encoded into a latent representation ft∈d0by

where fs=MLP(st) ∈dsand hT=RNN(Xt) ∈dtare respectively the output of MLP from stand the hidden representation of the last step of RNN from Xt, ‘⊕’ represents the vector concatenation, and g:ds+dt→dois the binary prediction function (layer), which is typically represented as fully-connected layer.

Further regarding block430, the encoded representation is evaluated by supervised metric learning loss, e.g., triplet loss:

where ·(+:=max(0,·), saq:=∥tanh(fa)−tanh(fq)∥(q ∈ {p,n}), and fa, fp, fnare features extracted by the dual-channel feature extractor220respectively from an anchor, positive and negative input samples. Each sample must have a time series segment and can also has static statuses.

Anchor samples are randomly selected from all data segments, positive samples are randomly selected from data samples which belongs to the same classes as anchors, and negative samples are randomly selected from data samples which belongs to different classes from anchors.

Further regarding block440, model parameters of the dual-channel feature extractor220are updated so that the loss function is smaller based on stochastic gradient descent.

Further regarding block450, if the stopping condition is not satisfied, then the training loop is repeated from block320, If stopping condition is satisfied, then the training stage is finished.

FIG.5is a flow diagram showing an exemplary hashing stage500, in accordance with an embodiment of the present invention.

At block510, extract, by the data preprocessor210, multivariate time series segments and corresponding static statuses from historical data.

At block520, jointly encode, by the dual-channel feature extractor220, all of the multivariate time series segments and the corresponding static statuses into compact representations.

At block530, convert, by the binary code extractor230, all compact representations into binary codes.

At block540, store the binary codes in the binary code database240.

Further regarding block510, multivariate time series segments and corresponding static status are extracted.

Further regarding block520, all of the multivariate time series segments with the corresponding static statuses are encoded by the dual-channel feature extractor220, which is trained in the training stage, into compact representations.

Further regarding block530, all compact representations obtained in block220are converted into binary vectors by checking the sign of all entries of the compact representations.

FIG.6is a flow diagram showing an exemplary retrieval stage600, in accordance with an embodiment of the present invention.

At block610, jointly encode, by the dual-channel feature extractor220, a current time series segment with static statuses into a compact representation.

At block620, convert, by the binary code extractor230, the compact representation into a binary code.

At block630, compute Hamming (or other) distances between the binary code and all binary codes in the binary code database240.

At block640, retrieve the top relevant multivariate time series segments based on the distance, where the shorter the distance, the more relevant the multivariate time series segments.

At block650, perform an action responsive to at least on the top relevant multivariate time series segments. For example, the top relevant multivariate times series segments can indicate an impending collision by a motor vehicle. In such a case, accident avoidance measures involving controlling one or more systems of a motor vehicle such as steering, braking, accelerating, stability, lighting, and so forth.

Further regarding block610, for a current observed time series segment with static statuses, a compact representation (feature) is extracted based on the dual-channel feature extractor220learned in the training stage.

Further regarding block620, the compact representation extracted in block610is converted into a binary code by checking signs of all entries in the feature vector embodying the compact representation.

Further regarding block630, the Hamming distances between the binary code converted in block320and all binary codes in the binary code database230are computed.

FIG.7is a block diagram showing an exemplary environment700to which the present invention can be applied, in accordance with an embodiment of the present invention.

In the environment700, a user788is located in a scene with multiple objects799, each having their own locations and trajectories. The user788is operating a vehicle772(e.g., a car, a truck, a motorcycle, etc.) having an ADAS777.

The ADAS777receives one or more of the top multivariate time series segments.

Responsive to the one or more of the top multivariate time series segments, a vehicle controlling decision is made. To that end, the ADAS777can control, as an action corresponding to a decision, for example, but not limited to, steering, braking, and accelerating systems.

Thus, in an ADAS situation, steering, accelerating/braking, friction (or lack of friction), yaw rate, lighting (hazards, high beam flashing, etc.), tire pressure, turn signaling, and more can all be efficiently exploited in an optimized decision in accordance with the present invention.

The system of the present invention (e.g., system777) may interface with the user through one or more systems of the vehicle772that the user is operating. For example, the system of the present invention can provide the user information through a system772A (e.g., a display system, a speaker system, and/or some other system) of the vehicle772. Moreover, the system of the present invention (e.g., system777) may interface with the vehicle772itself (e.g., through one or more systems of the vehicle772including, but not limited to, a steering system, a braking system, an acceleration system, a steering system, a lighting (turn signals, headlamps) system, etc.) in order to control the vehicle and cause the vehicle772to perform one or more actions. In this way, the user or the vehicle772itself can navigate around these objects799to avoid potential collisions there between. The providing of information and/or the controlling of the vehicle can be considered actions that are determined in accordance with embodiments of the present invention.

While described with respect to an ADAS, the present invention can be applied to a myriad of applications involving, e.g., a trajectory. For example, navigation involving automated agents, robots, assistive technologies for blind people, and/or so forth can be exploited by embodiments of the present invention.

FIG.8is a diagram showing exemplary time series801, time series segments802, and static statuses803, in accordance with an embodiment of the present invention.

The time series801represents the entire overall time series. The time series segments802represent various portions of the time series801. The static statuses803are system statuses that do not change within the same consecutive time points as time series segments802. The difference between static statuses803and time series801is the frequency of change. While time series801changes every moment, static statuses803change only at certain time intervals.